# Simple mouseray picking in openGL

Ive been looking at tutorials and trying to figure out how to do basic ray-picking. But I'm stuck at figuring out what space to do the distance calculations in. What space does glm::unproject() lead to exactly?

This is what I'm doing:

first I get the mouse unprojected, like so

//mouse ray start
vec3 m_uproj = glm::unProject(
vec3(mouse_xy .x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy .y*glutGet(GLUT_WINDOW_HEIGHT),0.0f),
workshop.access_gui()->view_mat(),
workshop.access_gui()->proj_mat(),
glm::ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

//end of ray
vec3 m_uproj2 = glm::unProject(
vec3(mouse_xy.x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy.y*glutGet(GLUT_WINDOW_HEIGHT), 1.0f),
workshop.access_gui()->view_mat(),
workshop.access_gui()->proj_mat(),
ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));


Then I find its direction, mray, like so

vec3 mouse_ray = normalize(m_uproj2 - m_uproj); //get mray direction


And Im expecting to find the closest point to some_object by using this calculation:

vec3 closest_point = mouse_ray * glm::dot(locations[i], mouse_ray); //closest point;


But locations seems to be in the wrong space? Or am I thinking about this the wrong way? Ive been looking around, but I cant find anywhere that explains just this part that I must be misunderstanding.

the idea is to compare the distance between closest point and locations[i], but the results are incorrect. Im getting something like this:

Where it should be red only if the cursor is over the square.

What space does glm::unproj() put my ray in anyway? And in what space should I put the objects that I want to pick/highlight?

• My guess: your cube is modelled at origin? You should pass as 2nd argument the modelview matrix, not just the view matrix. So multiply the view with the transformation matrix of the cube. Depending on glm convention you may need to premult or postmult. Not sure. – Bram Apr 18 '18 at 19:09
• no, the function is asking for the view matrix. that is not the solution (disregard the up arrow) – Charlie Apr 20 '18 at 5:05

The solution is this:

//world space, notice z=0.0f
vec3 mouse_world_nearplane = glm::unProject(
vec3(mouse.x*glutGet(GLUT_WINDOW_WIDTH), mouse.y*glutGet(GLUT_WINDOW_HEIGHT), 0.0f),
workshop.access<_gui>()->view_mat(), //view matrix
workshop.access<_gui>()->proj_mat(),
ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

//world space, notice z=1.0f
vec3 mouse_world_farplane = glm::unProject(
vec3(mouse.x*glutGet(GLUT_WINDOW_WIDTH), mouse.y*glutGet(GLUT_WINDOW_HEIGHT), 1.0f),
workshop.access<_gui>()->view_mat(), //view matrix
workshop.access<_gui>()->proj_mat(),
ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));


glm::unproject() takes:

a vec3 xy width/height and z, with z being plane/depth (0=close, 1=infinite)
your view_matrix (as given by glm::lookat)
your projection_matrix (as given by glm::perspective)
a viewport 0,0,width,height


And returns xyz as they would be in world space. World space is the same space that you would normally think in, where you get your object after applying the 3 initial transformations: [translate|rotate|scale] making up the objects identity matrix.

Meaning, the values returned by by glm::unproj() returns values that should be thought of in world space. So the solution is simple:

vec3 closest_point = glm::closestPointOnLine(
locations[i],
mouse_world_nearplane,
mouse_world_farplane);

if (glm::distance(locations[i], closest_point) < 0.1f)colors[i] = vec4(0.5f);
else colors[i] = vec4(1.0f);