Ive been looking at tutorials and trying to figure out how to do basic ray-picking. But I'm stuck at figuring out what space to do the distance calculations in. What space does glm::unproject() lead to exactly?

This is what I'm doing:

first I get the mouse unprojected, like so

//mouse ray start
vec3 m_uproj = glm::unProject(
    vec3(mouse_xy .x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy .y*glutGet(GLUT_WINDOW_HEIGHT),0.0f),
    glm::ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

//end of ray
vec3 m_uproj2 = glm::unProject(
    vec3(mouse_xy.x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy.y*glutGet(GLUT_WINDOW_HEIGHT), 1.0f),
    ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

Then I find its direction, mray, like so

vec3 mouse_ray = normalize(m_uproj2 - m_uproj); //get mray direction

And Im expecting to find the closest point to some_object by using this calculation:

vec3 closest_point = mouse_ray * glm::dot(locations[i], mouse_ray); //closest point;

But locations seems to be in the wrong space? Or am I thinking about this the wrong way? Ive been looking around, but I cant find anywhere that explains just this part that I must be misunderstanding.

the idea is to compare the distance between closest point and locations[i], but the results are incorrect. Im getting something like this:

enter image description here

Where it should be red only if the cursor is over the square.

What space does glm::unproj() put my ray in anyway? And in what space should I put the objects that I want to pick/highlight?

  • 1
    \$\begingroup\$ My guess: your cube is modelled at origin? You should pass as 2nd argument the modelview matrix, not just the view matrix. So multiply the view with the transformation matrix of the cube. Depending on glm convention you may need to premult or postmult. Not sure. \$\endgroup\$
    – Bram
    Apr 18, 2018 at 19:09
  • \$\begingroup\$ no, the function is asking for the view matrix. that is not the solution (disregard the up arrow) \$\endgroup\$
    – Charlie
    Apr 20, 2018 at 5:05

1 Answer 1


The solution is this:

//world space, notice z=0.0f
vec3 mouse_world_nearplane = glm::unProject(
    vec3(mouse.x*glutGet(GLUT_WINDOW_WIDTH), mouse.y*glutGet(GLUT_WINDOW_HEIGHT), 0.0f),
    workshop.access<_gui>()->view_mat(), //view matrix
    ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

//world space, notice z=1.0f
vec3 mouse_world_farplane = glm::unProject(
    vec3(mouse.x*glutGet(GLUT_WINDOW_WIDTH), mouse.y*glutGet(GLUT_WINDOW_HEIGHT), 1.0f),
    workshop.access<_gui>()->view_mat(), //view matrix
    ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

glm::unproject() takes:

a vec3 xy width/height and z, with z being plane/depth (0=close, 1=infinite)
your view_matrix (as given by glm::lookat) 
your projection_matrix (as given by glm::perspective)
a viewport 0,0,width,height

And returns xyz as they would be in world space. World space is the same space that you would normally think in, where you get your object after applying the 3 initial transformations: [translate|rotate|scale] making up the objects identity matrix.

Meaning, the values returned by by glm::unproj() returns values that should be thought of in world space. So the solution is simple:

vec3 closest_point = glm::closestPointOnLine(

if (glm::distance(locations[i], closest_point) < 0.1f)colors[i] = vec4(0.5f);
else colors[i] = vec4(1.0f);

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .