# From screen space to world space for a 2D game

I'm making a 2D game and I want the user to be able to position an object in the world using mouse input. After a while I found an answer to 'OpenGL Math - Projecting Screen space to World space coords' on Stack Overflow but even after working on it for multiple hours I couldn't get the right depth (z) value; after clicking the x and y coordinates of the square are correct for this z value, but as you can see the z value is way to big in this case and hence the size is too; It should be as big as all the other squares: (I clicked where the circle is).

My current handler is:

void mouse_button_callback(GLFWwindow *window, int button, int action, int mods)
{
if (button == GLFW_MOUSE_BUTTON_LEFT && action == GLFW_PRESS)
{
double x, y;
int width, height;
glfwGetCursorPos(window, &x, &y);
glfwGetWindowSize(window, &width, &height);

vec4 pos1(
static_cast<float>(x / (width / 2.f)) - 1,
1 - static_cast<float>(y / (height / 2.f)), 0 /*☃*/, 1);

pos1 = MVPinv * pos1;
pos1.w = 1 / pos1.w;
pos1.x *= pos1.w;
pos1.y *= pos1.w;
pos1.z *= pos1.w;

std::cout << pos1.x << " / " << pos1.y << " / " << pos1.z << " // " << pos1.w << std::endl;

objects->position = vec3(pos1);
}
}


Here MVPinv is the inverse of the model-view-projection matrix (don't mind the model matrix, it is the identity matrix and if I understand this correctly I can probably get rid of it in this place). I'm using modern OpenGL/GLEW, GLFW and GLM. Tweaking the z value indicated by ☃ can give me the correct result, but only for a certain view-projection matrix.

All objects I draw currently have a z value of 0. My question is thus: what is the best way to get the right xy coordinates if I want z to be 0?

Note: I'm very new to all these concepts of view-projection matrices, the w value, OpenGL and game development in general.

Edit: This is how the matrix is generated (more or less copied from a tutorial):

const mat4 Projection = perspective(radians(45.f), 4.0f / 3.0f, 0.1f, 100.0f);
const mat4 View = lookAt(
vec3(0, 0, 10),
vec3(0, 0, 0),
vec3(0, 1, 0)
);
const mat4 Model = mat4(1.0f);

MVP = Projection * View * Model;
MVPinv = inverse(MVP);


vec2 vecTranslate;

float factScale;
vec2 vecScale;

mat4 matTranslation, matScaling, matTransform;

void updateTranslate()
{
matTranslation = translate(mat4(1), vec3(vecTranslate, 0));
}

void updateScale(const int viewWidth, const int viewHeight)
{
vecScale = vec2(factScale, factScale / viewHeight * viewWidth);
matScaling = scale(mat4(1), vec3(vecScale, 1));
matTransform = matTranslation * matScaling;
}

void updateViewport(const int width, const int height)
{
glViewport(0, 0, width, height);
}

void mouse_button_callback(GLFWwindow *window, const int button, const int action, const int mods)
{
if (button == GLFW_MOUSE_BUTTON_LEFT && action == GLFW_PRESS)
{
double x, y;
int width, height;
glfwGetCursorPos(window, &x, &y);
glfwGetWindowSize(window, &width, &height);

const vec2 view(x / (width / 2.f) - 1,
1 - y / (height / 2.f));

objects->position = vec3((view - vecTranslate) / vecScale, 0);

/* Or:
const vec4 view(x / (width / 2.f) - 1,
1 - y / (height / 2.f), 0, 1);
const auto pos = inverse(matTransform) * view;
objects->position = {pos.x / pos.w, pos.y / pos.w, 0};
*/
}
}

void window_size_callback(GLFWwindow *window, const int width, const int height)
{
updateViewport(width, height);
}

• You're in 2D, so the full generality of a projection matrix might be more than you need. If your camera doesn't rotate, then the mapping between world & screen space should just be a scale (multiply) and a shift (add). We'll need to see how you're mapping your world coordinates to the screen to be able to tell you how to reverse the operation. – DMGregory Aug 3 '18 at 14:46
• @DMGregory I added the way I generated the matrices. Does the method that you proposed also work with matrices? Could you provide an example? – SWdV Aug 3 '18 at 14:57

You don't even need to use Look At and projection matrices if you are in 2D. They are good for 3D, but needless complexity for 2D. Your camera view can be simply the multiplication of a rotation matrix, scaling matrix and translation matrix. If your camera doesn't rotate, you can remove the rotation matrix out of it. Then your view matrix can be:

mat4 translation_matrix;
mat4 scaling_matrix;
mat4 view_matrix;

translation_matrix = mat4_translation(view_x, view_y, 0.0);
scaling_matrix  mat4_scaling(1.0 / view_width, 1.0 / view_height, 1.0);

view_matrix = translation_matrix * scaling_matrix;


Knowing where the player is clicking, in world coordinates, is a matter of adding view_x/y with cursor_x/y. No need to work with matrices here.

All objects I draw currently have a z value of 0. My question is thus: what is the best way to get the right xy coordinates if I want z to be 0?

The whole Z problem seems to me caused by the use of the projection/look at matrices.

• I struggled a bit to get everything right but in the end it finally seems to work, thanks! I'll add my updated code to the question for reference. I still have a question: why does inverse(view_matrix) not seem to work here? – SWdV Aug 4 '18 at 13:27
• I'm not sure why it's not working in your case, because calculating the matrix inverse is not something so complex. My only suspect is your "look at" or perspective matrix could not be adequate, as there are multiple ways of creating those, each with slight adjustments. – Ferreira da Selva Aug 4 '18 at 16:44
• For example, the code mat4 translate = glm::translate(mat4(1), vec3(1, 1, 0)), scale = glm::scale(mat4(1), vec3(2, 2, 0)), inv = glm::inverse(translate * scale); produces a matrix inv with -nan(ind) in every cell. Any idea what I'm doing wrong? – SWdV Aug 4 '18 at 20:14
• Try passing vec3(2, 2, 1) to the scale matrix, instead of vec3(2, 2, 0). NaN is result of a division by zero. As far as I know, it's caused by 0.0 / 0.0. – Ferreira da Selva Aug 4 '18 at 20:58
• Makes sense... And now using the inverse to reverse the operation works too as with these matrices z stays 0. Thanks! – SWdV Aug 4 '18 at 21:35