You can calculate the ray from camera for any pixel
px,py with the following formula:
ray.origin = camera.getPosition();
relativeX = px * 2 / screenWidth - 1; // [0, screenWidth] -> [-1, 1]
relativeY = py * 2 / screenHeight - 1; // [0, screenHeight] -> [-1, 1], might need to be flipped depending on the coordinate system
ray.direction = vector(relativeX * sin(FOV_X / 2),
relativeY * sin(FOV_Y / 2),
This creates the rays first with identity camera and then transforms it according to the camera orientation. If your camera is a 4x4 matrix,
camera.getPosition() should return the fourth column/row from the matrix and
camera.getOrientation() should return the 3x3 upper left part of the matrix. If you have ModelToView matrix, then camera matrix is the inverse of that. If the 3x3 part of your camera matrix has only rotation but no scaling, it's simply transpose of the matrix.
A more general approach is to invert the projection matrix also. For example vertex shader might be transforming coordinates with:
position = projection * worldToCamera * modelToWorld * vertexPosition;
modelToWorld matrix is not relevant in this case, so basically this is the same as:
position = projection * worldToCamera * worldPosition;
Which can be unprojected by
worldPosition = inverse(worldToCamera) * inverse(projection) * position;
worldPosition = inverse(projection * worldToCamera) * position;
If you unproject two vectors with different screen space z values (e.g.
px,py,1,1), you get two vectors in world space, which defines a line in 3D space for the ray you are looking for.