Simple mouseray picking in openGL

Question

Ive been looking at tutorials and trying to figure out how to do basic ray-picking. But I'm stuck at figuring out what space to do the distance calculations in. What space does glm::unproject() lead to exactly?

This is what I'm doing:

first I get the mouse unprojected, like so

//mouse ray start
vec3 m_uproj = glm::unProject(
    vec3(mouse_xy .x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy .y*glutGet(GLUT_WINDOW_HEIGHT),0.0f),
    workshop.access_gui()->view_mat(),
    workshop.access_gui()->proj_mat(),
    glm::ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

//end of ray
vec3 m_uproj2 = glm::unProject(
    vec3(mouse_xy.x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy.y*glutGet(GLUT_WINDOW_HEIGHT), 1.0f),
    workshop.access_gui()->view_mat(),
    workshop.access_gui()->proj_mat(),
    ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

Then I find its direction, mray, like so

vec3 mouse_ray = normalize(m_uproj2 - m_uproj); //get mray direction

And Im expecting to find the closest point to some_object by using this calculation:

vec3 closest_point = mouse_ray * glm::dot(locations[i], mouse_ray); //closest point;

But locations seems to be in the wrong space? Or am I thinking about this the wrong way? Ive been looking around, but I cant find anywhere that explains just this part that I must be misunderstanding.

the idea is to compare the distance between closest point and locations[i], but the results are incorrect. Im getting something like this:

Where it should be red only if the cursor is over the square.

What space does glm::unproj() put my ray in anyway? And in what space should I put the objects that I want to pick/highlight?

Answer 1

The solution is this:

//world space, notice z=0.0f
vec3 mouse_world_nearplane = glm::unProject(
    vec3(mouse.x*glutGet(GLUT_WINDOW_WIDTH), mouse.y*glutGet(GLUT_WINDOW_HEIGHT), 0.0f),
    workshop.access<_gui>()->view_mat(), //view matrix
    workshop.access<_gui>()->proj_mat(),
    ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

//world space, notice z=1.0f
vec3 mouse_world_farplane = glm::unProject(
    vec3(mouse.x*glutGet(GLUT_WINDOW_WIDTH), mouse.y*glutGet(GLUT_WINDOW_HEIGHT), 1.0f),
    workshop.access<_gui>()->view_mat(), //view matrix
    workshop.access<_gui>()->proj_mat(),
    ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));

glm::unproject() takes:

a vec3 xy width/height and z, with z being plane/depth (0=close, 1=infinite)
your view_matrix (as given by glm::lookat) 
your projection_matrix (as given by glm::perspective)
a viewport 0,0,width,height

And returns xyz as they would be in world space. World space is the same space that you would normally think in, where you get your object after applying the 3 initial transformations: [translate|rotate|scale] making up the objects identity matrix.

Meaning, the values returned by by glm::unproj() returns values that should be thought of in world space. So the solution is simple:

vec3 closest_point = glm::closestPointOnLine(
    locations[i], 
    mouse_world_nearplane, 
    mouse_world_farplane);

if (glm::distance(locations[i], closest_point) < 0.1f)colors[i] = vec4(0.5f);
else colors[i] = vec4(1.0f);