so i'm trying to make a mouse to world space function, and I seem to always end up with the rays cast offsetting.

I've looked at numerous tutorials, tried numerous examples. Tried with glm, gluUnProject, but what ever I try I always seem to get similar results to what my function gets. Which looks like something below: offset

Not sure which part of code to paste, so,

The function itself:

drx::util::V3 MTW(double z) {
    double w = drx::gfx::canvas.w; // 640.0;
    double h = drx::gfx::canvas.h; // 480.0;
    double mx = drx::view::window.mouse.x;
    double my = drx::view::window.mouse.y;
    double x = (2.0 * mx) / w - 1.0;
    double y = 1.0 - (2.0 * my) / h;
    drx::util::V4 clip = {x, y, z, 1};
    drx::util::MAT4F viewInv = this->GetView().Inve(); // inverted view matrix
    drx::util::MAT4F projectionInv = this->projection.Inve(); // inverted projection matrix
    drx::util::V4 wld = viewInv * projectionInv * clip;
    wld = wld.Divide(wld.w);
    drx::util::V3 ray = { wld.x, wld.y, wld.z };
    return ray;

With given z value set to 1.0, I get what was displayed in the gif, but if I set z to -1.0, I get a tiny line:

tiny line

View matrix:

this->view.matrix[0][0] = this->right.x;
this->view.matrix[0][1] = this->right.y;
this->view.matrix[0][2] = this->right.z;
this->view.matrix[1][0] = this->up.x;
this->view.matrix[1][1] = this->up.y;
this->view.matrix[1][2] = this->up.z;
this->view.matrix[2][0] = this->front.x;
this->view.matrix[2][1] = this->front.y;
this->view.matrix[2][2] = this->front.z;
this->view.matrix[0][3] = -this->position.Dot(this->right);
this->view.matrix[1][3] = -this->position.Dot(this->up);
this->view.matrix[2][3] = -this->position.Dot(this->front);
this->view.matrix[3][3] = 1.0f;

Projection matrix:

float sf = tanf(fov * 0.5);
this->matrix[0][0] = 1.0f / (ar * sf);
this->matrix[1][1] = 1.0f / sf;
this->matrix[2][2] = -((f + n) / (f - n));
this->matrix[3][2] = -1.0f;
this->matrix[2][3] = -((2.0f * f * n) / (f - n));

The lines in the image and gif, mean, red == camera.front, blue == camera.up and green == camera.right vectors.

I'm not too sure what sort of lines I should be seeing, so if anyone can shed some light into all that, I'd be very grateful. Will provide more info, if asked.

  • \$\begingroup\$ Try drx::util::V4 wld = projectionInv * clip; wld = wld.Divide(wld.w); wld = viewInv * wld; \$\endgroup\$
    – gotanod
    Commented May 31 at 7:35
  • \$\begingroup\$ @gotanod Same result as in the gif. \$\endgroup\$
    – Valtsuh
    Commented May 31 at 8:00
  • \$\begingroup\$ when you click on the screen, it should produce a ray between near and far planes, but it will be just a point in the screen. Only after moving the camera, you will be able to see the ray segment. The ray goes from z=-1 to z=1 (whole frustrum of the camera when you calculated it). So you draw a line between two points after converting the clip points (x,y,-1,1) and (x,y,1,1). \$\endgroup\$
    – gotanod
    Commented May 31 at 9:49

1 Answer 1


Answering my own question.

I was able to solve the problem, after all the head-scratching. The ray code itself is actually fine, where the error was, was window creation, I had:

CreateWindow( ... x, y, w, h, ...);

Which creates the windows outer size to be the given width(w) and height(h) values. I added:

RECT rect;
rect.left = 0;
rect.right = w;
rect.top = 0;
rect.bottom = h;
AdjustWindowRect(&rect, WS_CAPTION | WS_MINIMIZEBOX | WS_SYSMENU, false);
int nw = rect.right - rect.left;
int nh = rect.bottom - rect.top;
CreateWindow( ... x, y, nw, nh, ...);

Which calculates size for the inner window (or context area, excluding title bar etc.), to match the given w and h values, gave it a go, and that was it, problem solved.



You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .