# Picking object using ray tracer in opengl using inverses

I have a scene with many objects and I'd like to be able to pick them out using ray tracing. I used to have just model transformations happening in the vertex shader and I was able to pick out the objects by doing

        Vector4f p_screen(xpos,height-1-ypos,0,1);
Vector4f p_canonical((p_screen[0]/width)*2-1,(p_screen[1]/height)*2-1,0,1);
p_world = model.inverse() * p_canonical;
// Convert screen position to world coordinates
float xworld = p_world[0];
float yworld = p_world[1];

Vector4f ray_origin = Vector4f(xworld, yworld, 1, 1);
Vector4f ray_direction = Vector4f(0, 0, -1, 0);


That all worked perfectly.

However, I've just implemented a projection and view transformation into the vertex shader like such

    "#version 150 core\n"
"in vec3 position;"
"in vec3 normals;"

"out vec3 FragPos;"
"out vec3 Normal;"

"uniform mat4 model;"
"uniform mat4 view;"
"uniform mat4 proj;"

"void main()"
"{"
"    FragPos = vec3(model * vec4(position, 1.0));"
"    Normal = mat3(transpose(inverse(model))) * normals;"
"    gl_Position = proj * view * vec4(FragPos, 1.0);"
"}";


I tried to update my ray tracer to include the inverses of the view and projection matrices to be

 p_world = model.inverse() * view.inverse() * projection.inverse() * p_canonical;


but it's totally broken now and I'm not quite sure what I'm doing wrong. I know the view and projection matrices are working as it shows up on the window projected with perspective projection.

• Order of multiplication matters. Multiply your matrices in reverse first, then inverse resulting matrix, then transform your point. – O c e l o t Nov 28 '18 at 15:17