I'm trying to convert screen coordinate to world coordinate on mouse click event. For LWJGL 3 there's not GLU utility class is available whereas LWJGL 2 has.

I'm using JOML math classes and wrote following code, but its returning wrong world coordinate, I'm doing something wrong and couldn't figure out.

On program init, I get viewProjMatrix and viewport:

viewProjMatrixUniform = glGetUniformLocation(this.program, "viewProjMatrix");
IntBuffer viewportBuffer = BufferUtils.createIntBuffer(4);
int[] viewport = new int[4];
glGetIntegerv(GL_VIEWPORT, viewportBuffer);

On render loop, I calculate viewProjMatrix:

    .setPerspective((float) Math.toRadians(30), (float) width / height, 0.01f, 500.0f) // define min and max planes
    .lookAt(eye_x, eye_y, eye_z,
            eye_x, eye_y, 0.0f,
            0.0f, 2.0f, 0.0f);
glUniformMatrix4fv(viewProjMatrixUniform, false, viewProjMatrix.get(matrixBuffer));

I convert screen coordinate to world coordinate with following code:

DoubleBuffer mouseXBuffer = BufferUtils.createDoubleBuffer(1);
DoubleBuffer mouseYBuffer = BufferUtils.createDoubleBuffer(1);
glfwGetCursorPos(window, mouseXBuffer, mouseYBuffer);
double x = mouseXBuffer.get(0);
double y = mouseYBuffer.get(0);
System.out.println("clicked at: " + x + " - " + y);

Vector3f v3f = new Vector3f();
viewProjMatrix.unproject((float)x, (float)y, 0f, viewport, v3f);
System.out.println("world coordinate: " + v3f.x + " - " + v3f.y);

Here's the full source code: https://gist.github.com/digz6666/48bb433c83801ea4b82fa194f05b4f02


1 Answer 1


There are to steps to converting cursor coordinates to world coordinates: 1) Convert the mouse coords to a direction. 2) Perform something like a raycast to find to world coordinate.

Step 1: First you will need to convert mouse coords to a direction. This is a simple illustration of how you could do this: Conversion Sketch

(Source: http://antongerdelan.net/opengl/raycasting.html)

I also recommend taking a look at this tutorial: https://m.youtube.com/watch?list=PLRIWtICgwaX0u7Rf9zkZhLoLuZVfUksDP&v=DLKN0jExRIM. You should be able to use the math classes with LWJGL 3, so you shouldn't have a problem.

Step 2: This part is the hardest out of the two, if you don't already have some sort of physics engine up and running that supports raycasting. If you don't have one available, then it shouldn't be that difficult if you know what you are doing. Try searching online for tutorials. I really like video tutorials, so I recommend spending some time one YouTube. There are a lot of tutorials over there that are really good.

  • \$\begingroup\$ I'm trying to convert in just 2D space, is it raycasting necessary? \$\endgroup\$
    – digz6666
    Commented Jul 12, 2017 at 4:53
  • \$\begingroup\$ Not necessarily! Instead of doing the steps on the post, you could draw an image to a FBO, where each entity has a unique color. Then, get the color of the texture at the coordinates of the cursor, and find the entity with that color. If you want to know depth as well, you can do the same but with a depth buffer instead! But, this is heavily based on your rendering engine, and it could very well decrease performance, if you have a bad rendering engine. However, if you have a good rendering engine then it could increase performance and save you a lot of time. \$\endgroup\$
    – J22o
    Commented Jul 12, 2017 at 7:29

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .