I have a scene with many objects and I'd like to be able to pick them out using ray tracing. I used to have just model transformations happening in the vertex shader and I was able to pick out the objects by doing

        Vector4f p_screen(xpos,height-1-ypos,0,1);
        Vector4f p_canonical((p_screen[0]/width)*2-1,(p_screen[1]/height)*2-1,0,1);
        p_world = model.inverse() * p_canonical;
        // Convert screen position to world coordinates
        float xworld = p_world[0]; 
        float yworld = p_world[1]; 

        Vector4f ray_origin = Vector4f(xworld, yworld, 1, 1);
        Vector4f ray_direction = Vector4f(0, 0, -1, 0);

That all worked perfectly.

However, I've just implemented a projection and view transformation into the vertex shader like such

    "#version 150 core\n"
            "in vec3 position;"
            "in vec3 normals;"

            "out vec3 FragPos;"
            "out vec3 Normal;"

            "uniform mat4 model;"
            "uniform mat4 view;"
            "uniform mat4 proj;"

            "void main()"
            "    FragPos = vec3(model * vec4(position, 1.0));"
            "    Normal = mat3(transpose(inverse(model))) * normals;"
            "    gl_Position = proj * view * vec4(FragPos, 1.0);"

I tried to update my ray tracer to include the inverses of the view and projection matrices to be

 p_world = model.inverse() * view.inverse() * projection.inverse() * p_canonical;

but it's totally broken now and I'm not quite sure what I'm doing wrong. I know the view and projection matrices are working as it shows up on the window projected with perspective projection.

  • \$\begingroup\$ Order of multiplication matters. Multiply your matrices in reverse first, then inverse resulting matrix, then transform your point. \$\endgroup\$ – O c e l o t Nov 28 '18 at 15:17

Your Answer

By clicking "Post Your Answer", you acknowledge that you have read our updated terms of service, privacy policy and cookie policy, and that your continued use of the website is subject to these policies.

Browse other questions tagged or ask your own question.