Getting the Ray position from View and Projection Matrix

Question

I'm having some trouble calculating the direction and position of the ray from my matrices. I have tried some things such as:

private Vec3 getPick(Mat4 projection, Mat4 view) {
     Mat4 inverseProject = projection.copy().inverse();
     Mat4 pick = inverseProject.mul(view.copy());
     Vec3 pos = new Vec3(pick.m10, pick.m11, pick.m12);
     return pos;
}

I am using custom math classes and I'm pretty sure they work. Also, would it be better to perform this math in my shader? For better performance etc..?

I contacted Notch (creator of Minecraft) on this issue and he told me to inverse the view Matrix and multiply by the projection, then multiply by the mouse coordinates. I tried that, it didn't do much.

EDIT: Tried normalising the screen space. It seems to do the job except in negative directions. Plus its a little off.

Vec3 getPick(double w, double h, double ar) {
    double sx = ((Mouse.getX() / w / 2.0) - 1.0) * ar;
    double sw = 1.0 - (2 * ((h - Mouse.getY()) / h));

    Mat4 viewProjInv = view.copy().mul(projection).inverse();
    Vec3 mouseNormal = new Vec3(sx, sw, 1.0);
    Vec3 dir = viewProjInv.mul(mouseNormal);
    return dir.mul(1);
}

Answer 1

There are several steps you need to follow that will allow you to pick in a 3D application.

To transform screen coordinates to world space coordinates, you need to denormalize the scree-space coordinates.

Then, you need to multiply the the point of the cursor (assumed object doing the picking) in normalized device space with this matrix.

Then we would need to solve for the device point in regards to normalized space by solving it from the equation that gives us the screen point and viewport dimensions. This would be the origin of the raycast.

See, the thing is, you are half way there, but you are at the last half with the code supplied.

And then, after all of this, you construct your ray and need to do an intersection test with a bounding volume or mesh triangle.

In regards to doing the math in your shader, most likely not, but for something like that you could try making tests to see which is faster. However, rule #1 is don't worry about optimization until you need to. Rule #2 is never optimize.

I have not much experience with lwjgl, but is there a function that you can call that will transform mouse coordinates for you at all and return the vector? Also, you should show how you are calculating the direction as well as the mouse position.

I usually like to answer with source code, but I found this lwjgl java example on how to pick with OpenGL in 3D