Ive been looking at tutorials and trying to figure out how to do basic ray-picking. But I'm stuck at figuring out what space to do the distance calculations in. What space does glm::unproject()
lead to exactly?
This is what I'm doing:
first I get the mouse unprojected, like so
//mouse ray start
vec3 m_uproj = glm::unProject(
vec3(mouse_xy .x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy .y*glutGet(GLUT_WINDOW_HEIGHT),0.0f),
workshop.access_gui()->view_mat(),
workshop.access_gui()->proj_mat(),
glm::ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));
//end of ray
vec3 m_uproj2 = glm::unProject(
vec3(mouse_xy.x*glutGet(GLUT_WINDOW_WIDTH), mouse_xy.y*glutGet(GLUT_WINDOW_HEIGHT), 1.0f),
workshop.access_gui()->view_mat(),
workshop.access_gui()->proj_mat(),
ivec4(0, 0, glutGet(GLUT_WINDOW_WIDTH), glutGet(GLUT_WINDOW_HEIGHT)));
Then I find its direction, mray, like so
vec3 mouse_ray = normalize(m_uproj2 - m_uproj); //get mray direction
And Im expecting to find the closest point to some_object
by using this calculation:
vec3 closest_point = mouse_ray * glm::dot(locations[i], mouse_ray); //closest point;
But locations seems to be in the wrong space? Or am I thinking about this the wrong way? Ive been looking around, but I cant find anywhere that explains just this part that I must be misunderstanding.
the idea is to compare the distance between closest point and locations[i], but the results are incorrect. Im getting something like this:
Where it should be red only if the cursor is over the square.
What space does glm::unproj()
put my ray in anyway? And in what space should I put the objects that I want to pick/highlight?