3D Picking: Doesn't Work Consistently

I'm trying to implement a 3D picking function that at the moment simply calculates the ray being sent from the camera after the user clicks the mouse on the window.

It kind of works well when the camera is facing forward (orientation [0.0, 0.0, 1.0]) but when I orient the camera upside-down or looking up/down it produces really stupid results (i.e. the ray is wildly out-of-line with the cursor, or even sometimes points behind the camera somewhere).

Here is the code:

// sv (the SceneViewport) is the window/viewport I'm clicking in
// pixel (the Vector2) is the position of the cursor, relative to the viewport centre, in pixels
public Ray PixelRayCast(SceneViewport sv, Vector2 pixel) {
Vector2 viewportSizePixels = sv.SizePixels;
float aspectRatio = sv.AspectRatio;
float invFov = (float) Math.Tan(GetVerticalFOV(aspectRatio) * 0.5f);
Vector3 rayOrientation = new Vector3(
((2f * pixel.X) / viewportSizePixels.X) / (1f / (aspectRatio * invFov)),
((2f * pixel.Y) / -viewportSizePixels.Y) / (1f / invFov),
1f
) * Quaternion.FromVectorTransition(Vector3.FORWARD, Orientation); // Orientation is the camera's current orientation
Vector3 rayStart = Position; // Position is the camera's current position
return new Ray(rayStart, rayOrientation);
}


I'm not really a great mathematician so I'm hoping someone will be able to read through my small function and spot the mathematical error I'm making.

• It might help if you visualize the picking ray in 3d space while you move your camera around. You might find that its something silly and easily fixed, such as the x axis is going the opposite direction that it should, but everything else is working correctly. – Alan Wolfe Jul 13 '15 at 17:02
• I'll second the visualization suggestion. Raypicking was a huge pain until I wrote a small primitive drawing system. Being able to visualize Rays makes writing a raypicking solution bearable. – Honeybunch Jul 13 '15 at 22:20
• I 3rd. Draw debug geometry. – Seth Battin Jul 14 '15 at 1:24