Project camera-to-pixel ray in world coordinates

Question

I'm trying to implement a very simple ray tracing function but I have a problem with projecting the camera ray (from camera to pixel) to the world space.

In the screen coordinates have the pixel position (between -1,-1 and 1,1), the field of view (45°). My course tells me that the direction of my ray is

vec3 screenDir = (screen_x, screen_y, -1/tan(fovy)) = (screen_x, screen_y, -1)

So this should give me the direction of the camera ray in screen coordinates.

After that I multiply the direction by the inverse of my ModelViewProjection matrix and divide by the w component:

vec4 worldDir = inverse(MVP) * vec4(screenDir, 1.0); 
worldDir /= worldDir.w;
vec3 d = normalize(worldDir.xyz);

In theory this should be my ray direction in world space.

I have implemented a small plane-intersection formula to check if this works but the result is completely false (I get a small circle that stays bellow my camera)

float t = dot((cam_pos - pointOnPlane), planeNormal) / dot(d, planeNormal);
vec3 intersection = cam_pos + t * d;
vec4 test = MVP * vec4(intersection, 1.0);
test /= test.w;
if(distance(screen_coord, test.xy)<0.1)
    color = vec3(1.0);

Is there a problem with my code ? I've read so many tutorial and courses and forum posts and I don't understand why my code doesn't work...

Answer 1

If I understand your question: Your asking how to get the world coordinates of the camera, with a look vector to raytrace?

Start by getting the position of your camera, which is simple how you translated the world to get from it's position relative to 0,0 (where the camera always is). Then, look at how you rotate the scene to do the same. From there, you have angle and position of the world from the camera. This is also a normal vector to the screen plane. To raytrace/get a look vector, just use some trigonometry.