Cursor position to a 3D ray using angles

I've been stuck for a month trying to get gluUnProject working. After my attempts to use gluUnProject failed (as well as attempts to implement gluUnProject functionality manually) I implemented method to get ray using angles.

float RadToDeg = 180.0f / (float)M_PI; // conversion coefficient to degree

float DeltaX = tan(fovy/2 * DegToRad) * 2 * (Width / 2 - winX) * zNear / Height;
float DeltaY = tan(fovy/2 * DegToRad) * 2 * (Height / 2 - winY) * zNear / Height;
float AngleX = atan(DeltaX / zNear) * RadToDeg;
float AngleY = atan(DeltaY / zNear) * RadToDeg;

float DeltaXY = sqrt(DeltaX*DeltaX + DeltaY*DeltaY);
float AngleXY = atan(DeltaXY / zNear) * RadToDeg;
float AxisAngle = atan(DeltaX / DeltaY) * RadToDeg;
vec3 RotationAxis = rotate(Camera->X, AxisAngle, Camera->Z);
if (DeltaY < 0) { RotationAxis = -RotationAxis; }

vec3 ray = rotate(-(Camera->Z), AngleXY, RotationAxis); //getting ray direction
ray = vec3(ray.x * scale, ray.y * scale, ray.z * scale); //scale ray if needed

vec3 RayEndPoint = Camera->Position + ray;

The main idea behind my method is to rotate ray pointing perpendicularly from the middle of the screen (like in first person shooters) to position where you clicked.

Necessary values are fovy angle (field of view in vertical direction), zNear (distance from Camera to near plane) and Camera Transformation Matrix(Camera->X, Camera->Y, Camera->Z, Camera->Position - 1st, 2nd, 3rd and 4th columns). Difference between Camera Transformation Matrix and View Matrix here.

Besides we need some method rotate(vec3 vector,float angle,vec3 axis) that will rotate any vector at any angle around any arbitrary axis. Explanation how it works will consume too many lines (check photo of my calculations). Q: Is any easier or better way to do it? Any information or links welcomed.

• Yes, there's an easier way: fix your bugs with the matrix version. That's the easiest way to do it, by far. I promise. :) Sep 7 '15 at 18:44

A month ago I came across a youtube video which explains how to do it, although this video is a tutorial for Java, you will probably be able to use it.

The video was based on this text article.

I hope this is usefull for you.

PS: This tutorial uses the LWJGL Matrix functions, this StackExchange answer has some code which you will be able to copy if you need the inverse of a matrix.

• I've already implemented such approach and explained here. What I was asking in current thread is how to get ray using angle deviations from other ray, e.g. get ray 30 degrees left and 45 degrees down from some ray we have.
– mioe
Sep 7 '15 at 18:53
• @mioe So you are asking for a 3D vector multiplication? Sep 7 '15 at 19:55

Just multiply a clip-space coordinate (x and y are screen-coords ranging from [-1;1], z=0, w=1) with the inverse of the view-projection matrix. Then you need to divide the resulting homogenous vector with its w-component. This gives you a position on the desired ray. To get the ray direction just subtract the camera position and normalize the result.

While this is surely not the best solution in number of arithmetical operations, I find it super intuitive. Since it relies on function which are included in most 3D-math libraries, it is usually also super short in its implementation.

• I just realized that this is roughly what gluUnProject is supposed to do. opengl.org/sdk/docs/man2/xhtml/gluUnProject.xml Sep 6 '15 at 20:14
• I tried that already see my other question with no success. I undestand that it should work - so many topics and examples, but I'm missing something.
– mioe
Sep 6 '15 at 20:16