Cursor position to a 3D ray using angles

Question

I've been stuck for a month trying to get gluUnProject working. After my attempts to use gluUnProject failed (as well as attempts to implement gluUnProject functionality manually) I implemented method to get ray using angles.

 float RadToDeg = 180.0f / (float)M_PI; // conversion coefficient to degree
 float DegToRad = 1 / RadToDeg; // conversion coefficient to radians 

 float DeltaX = tan(fovy/2 * DegToRad) * 2 * (Width / 2 - winX) * zNear / Height;
 float DeltaY = tan(fovy/2 * DegToRad) * 2 * (Height / 2 - winY) * zNear / Height;
 float AngleX = atan(DeltaX / zNear) * RadToDeg;
 float AngleY = atan(DeltaY / zNear) * RadToDeg;

 float DeltaXY = sqrt(DeltaX*DeltaX + DeltaY*DeltaY);
 float AngleXY = atan(DeltaXY / zNear) * RadToDeg;
 float AxisAngle = atan(DeltaX / DeltaY) * RadToDeg;
 vec3 RotationAxis = rotate(Camera->X, AxisAngle, Camera->Z);
 if (DeltaY < 0) { RotationAxis = -RotationAxis; }

 vec3 ray = rotate(-(Camera->Z), AngleXY, RotationAxis); //getting ray direction
 ray = vec3(ray.x * scale, ray.y * scale, ray.z * scale); //scale ray if needed

 vec3 RayEndPoint = Camera->Position + ray;

The main idea behind my method is to rotate ray pointing perpendicularly from the middle of the screen (like in first person shooters) to position where you clicked.

Necessary values are fovy angle (field of view in vertical direction), zNear (distance from Camera to near plane) and Camera Transformation Matrix(Camera->X, Camera->Y, Camera->Z, Camera->Position - 1st, 2nd, 3rd and 4th columns). Difference between Camera Transformation Matrix and View Matrix here.

Besides we need some method rotate(vec3 vector,float angle,vec3 axis) that will rotate any vector at any angle around any arbitrary axis. Explanation how it works will consume too many lines (check photo of my calculations).

Q: Is any easier or better way to do it? Any information or links welcomed.

Answer 1

Just multiply a clip-space coordinate (x and y are screen-coords ranging from [-1;1], z=0, w=1) with the inverse of the view-projection matrix. Then you need to divide the resulting homogenous vector with its w-component. This gives you a position on the desired ray. To get the ray direction just subtract the camera position and normalize the result.

While this is surely not the best solution in number of arithmetical operations, I find it super intuitive. Since it relies on function which are included in most 3D-math libraries, it is usually also super short in its implementation.

Answer 2

A month ago I came across a youtube video which explains how to do it, although this video is a tutorial for Java, you will probably be able to use it.

The video was based on this text article.

I hope this is usefull for you.

PS: This tutorial uses the LWJGL Matrix functions, this StackExchange answer has some code which you will be able to copy if you need the inverse of a matrix.