0
\$\begingroup\$

I'm trying to implement screen space to world space in my engine to allow object selection in editor. I understand that the process should essentially be the reverse of projection. So to go from world space to screen space, I would do ModelMatrix * ViewMatrix * ProjectionMatrix * Point. So to go from screen space to world space I should be able to do Point * (ProjectionMatrix * ViewMatrix * ModelMatrix)^-1. I've been assuming that the ModelMatrix should always be the identity matrix.

The code I have works perfectly when the camera is at the origin, but when I start moving the camera around, the further away from the origin it gets, the more incorrect the results become. I should note that m_currentCamera->GetViewMatrix() flips the z axis so the camera is looking at +Z instead of negative Z, and also inverts the camera transform matrix as necessary to convert to camera space.

glm::mat4 projection_matrix = glm::perspective(glm::radians(60.), (double)g_width / (double)g_height, (double)0.1, (double)10000);
projection_matrix[1][1] *= -1;


glm::mat4 view_matrix = m_currentCamera->GetViewMatrix();

// Cast mouse position from screen space to world space
POINT mouse_position = InputState::GetMousePosition();

float mouse_x = mouse_position.x;
float mouse_y = mouse_position.y;

// Convert from screen space to Vulkan NDC space
mouse_x = (2.f * (mouse_x / (float)g_width)) - 1.f;
mouse_y = (2.f * (mouse_y / (float)g_height)) - 1.f;
        
glm::vec4 screen_space_mouse = glm::vec4(mouse_x, mouse_y, 0.f,  1);

        
glm::mat4 pv =  projection_matrix * view_matrix;
screen_space_mouse =  glm::inverse(pv) * screen_space_mouse;


screen_space_mouse /= screen_space_mouse.w;
m_world_space_mouse = screen_space_mouse;

What am I missing? The behavior is a little strange, when the camera is in the -Z, the ray ends up pointing behind the camera. I've done a ton of research on screen space unproject and I can't see a difference in what I'm seeing online vs what my code is doing.

\$\endgroup\$

1 Answer 1

0
\$\begingroup\$

Well I figured it out. The screen space mouse needed to be at z of 1 like this:

glm::vec4 screen_space_mouse = glm::vec4(mouse_x, mouse_y, 1.f,  1);

I thought I had tried at z of 1 but I must have missed that case in my testing.

\$\endgroup\$

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .