OpenGL - have object follow mouse

Question

I want to have an object follow around my mouse on the screen in OpenGL. (I am also using GLEW, GLFW, and GLM). The best idea I've come up with is:

Get the coordinates within the window with glfwGetCursorPos.

The window was created with

window = glfwCreateWindow( 1024, 768, "Test", NULL, NULL);

and the code to get coordinates is

double xpos, ypos;
glfwGetCursorPos(window, &xpos, &ypos);

Next, I use GLM unproject, to get the coordinates in "object space"

glm::vec4 viewport = glm::vec4(0.0f, 0.0f, 1024.0f, 768.0f);
glm::vec3 pos = glm::vec3(xpos, ypos, 0.0f);
glm::vec3 un = glm::unProject(pos, View*Model, Projection, viewport);

There are two potential problems I can already see. The viewport is fine, as the initial x,y, coordinates of the lower left are indeed 0,0, and it's indeed a 1024*768 window. However, the position vector I create doesn't seem right. The Z coordinate should probably not be zero. However, glfwGetCursorPos returns 2D coordinates, and I don't know how to go from there to the 3D window coordinates, especially since I am not sure what the 3rd dimension of the window coordinates even means (since computer screens are 2D). Then, I am not sure if I am using unproject correctly. Assume the View, Model, Projection matrices are all OK. If I passed in the correct position vector in Window coordinates, does the unproject call give me the coordinates in Object coordinates? I think it does, but the documentation is not clear.

Finally, to each vertex of the object I want to follow the mouse around, I just increment the x coordinate by un[0], the y coordinate by -un[1], and the z coordinate by un[2]. However, since my position vector that is being unprojected is likely wrong, this is not giving good results; the object does move as my mouse moves, but it is offset quite a bit (i.e. moving the mouse a lot doesn't move the object that much, and the z coordinate is very large). I actually found that the z coordinate un[2] is always the same value no matter where my mouse is, probably because the position vector I pass into unproject always has a value of 0.0 for z.

Edit: The (incorrectly) unprojected x-values range from about -0.552 to 0.552, and the y-values from about -0.411 to 0.411.

Answer 1

Disclaimer: I generally don't work in 3D, so possible inaccuracies, vague details, and untested thoughts may follow.

When you say you want the object to "follow the mouse", what do you really mean? In another sense, you can interpret it like this: you want a certain point on the object to be "below" the mouse at all times. We'll call this point the "anchor point" of the object; moving the anchor point moves all the vertices of the object. How can we move this point in a way that it always lies under the mouse? To answer that, we need to take a look at the viewing frustum:

Some plane cutting the view frustum (out farther than the near view plane but closer than the far view plane) is what you'd normally think of as the screen - i.e. where the mouse would be. The exact place this plane is located is defined by your field of view. Objects are projected onto this plane and the plane is drawn onto the screen. That means, then, that the mouse lies on this plane in 3D space.

Consider this: If you were to draw a line from the camera's position to the mouse's position on the near plane, that line (extended infinitely) consists of all the points under the mouse. This implies, then, that for our anchor point to always be under the mouse, the line I just described must always intersect that point.

The problem is that the line has infinite points; where along that line should the anchor point be placed? The intuitive answer is that "the distance from the mouse to that point should always remain the same", but this is not correct; if that were true, the object would move along a sphere, and the viewing frustum is made of planes. Ah - therein lies our answer! We want the object to move along a plane, not along a circle.

So, then, we have two constraints:

1. We want the anchor point to move along the plane which is parallel to the near/far planes and which contains the original anchor point.
2. **We want the anchor point to always lie on the line defined by the mouse's location in 3D space and the position of the camera. **

And the intersection of a line and a plane is a point - the new position of the object's anchor point. Having now defined the problem more precisely, we can get a high-level algorithm:

1. Pick the anchor point p_a on the object.
2. On a mouse move from p_1 to p_2 (3D coords), do the following:
3. Get the plane P parallel to the near/far planes which contains p_a.
4. Intersect the line defined by the camera and p1 with P. Call the intersection point x_1.
5. Intersect the line defined by the camera and p2 with P. Call the intersection point x_2.
6. Move each vertex of the object by the vector x_2 - x_1.

Obviously, I've left out a whole lot of details out, such as how to get the mouse position in 3D coords (which, incidentally, is what I think glm::unproject is doing right now), but most of those are mathematical in nature. Hopefully this gives you enough details to go off of.

---

After sleeping and rereading the question, I see I have barely answered the original questions posed. Here is what glm::project does, largely paraphrased from this presentation and this question: it takes a point on the screen and translates it into a point on the line underneath it, not to where the screen would be. This is what the Z coordinate is passed in for; it is a real between 0.0 and 1.0 representing the plane between the near and far view planes on which the point you receive will lie. So, right now, you are obtaining the position of the mouse on the near clip plane, which is probably much closer to the camera than you would like.

In terms of my original answer, you should instead set this Z coordinate to wherever between the clip planes your original anchor point is. Then, the points you will receive are exactly the points you would get from manually doing steps (4) and (5).

The likely reason the Z coordinate is always the same is that you have your camera pointed along the Z axis (i.e. the view planes are perpendicular to that axis); then, any point a plane parallel to the near/far planes is going to have the same Z coordinate.