# How to convert screen to world coordinates while using gluLookAt / gluPerspective or similar matrix transforms?

I am just starting an adventure in looking under the hood of graphics for a game project I've been working on for a while, and I could use some guidance.

I am using Python/Kivy (though that is not part of the concern), and am trying to use projection and modelview matrices to perform screen-to-world coordinate conversion. I am using something similar to gluLookAt and gluPerspective matrix transforms for those.

The issue I'm running into, is that the coordinates I get out of multiplying the mv and p matrices together and inverting them, then multiplying by NDC screen coords, the resulting coordinates are only either a fraction of a pixel away from the world position look_at is currently centered on, or a max of a few pixels +/- that point.

I know I'm missing something, and I would love if someone could help me understand. I wrote a standalone example gist, and made a short youtube video showing what problem I'm having.

The code in question, is the screen_to_world method in the gist, when using the camera_look_at_perspective method to create the MVP, which I will list here:

p = Matrix()
p.perspective(90., 16 / 9, 1, 1000)

self.canvas['projection_mat'] = p
self.canvas['modelview_mat'] = Matrix().look_at(w_x, w_y - 30, self.camera_scale * 350, w_x, w_y, 0, 0, 1, 0)


That is for creating the matrices, and...

def screen_to_world(self, x, y):
proj = self.canvas['projection_mat']
model = self.canvas['modelview_mat']

# get the inverse of the current matrices, MVP
m = Matrix().multiply(proj).multiply(model)
inverse = m.inverse()
w, h = self.size

# normalize pos in window
norm_x = x / w * 2.0 - 1.0
norm_y = y / h * 2.0 - 1.0

p = inverse.transform_point(norm_x, norm_y, 0)
# print('convert_from_screen_to_world', x, y, p)
return p[:2]


Was originally written to convert coordinates when using the previous projection matrix built using translate and creating a clip space (also included in the example).

While the implementation appears to be specific to Kivy, it is just a modelview matrix and projection matrix being used, and their Matrix.transform_point method used above is the same as multiplying a vec against the matrix in question. It can also include what appears to be the W part of a vec4, which I have also experimented with, with no apparent change.

Here is a screenshot of the standalone example, painting where I have moved the mouse on the screen (red) and where the resulting world coordinate ends up being (green). The goal is for the converted coordinates in the world to fall directly under the red.