How to pick a 3D object with known mouse coordinates if the camera is rotated 60 degrees?

Question

I am making a simple game engine for a simple RTS game and would like to be able to pick a cube and move it around with the mouse coordinates. The problem is that my camera is rotated 60 degrees just like any RTS game and I am not sure how to use gluUnProject..

Thanks in advance!!

Answer 1

Fundamentally, "what part of the world is under my cursor" is actually a problem of collision detection -- in terms of code, there's no difference between checking what's under the cursor, and checking what part of the world was hit by a fired bullet.

So, if we think of checking what's under our mouse cursor as 'firing a bullet', we're clearly firing it from the camera. And if you imagine that your screen is (for example) floating a meter in front of the camera, the bullet is being aimed to pass through a specific pixel on the virtual screen -- the pixel where the cursor is being drawn on the real screen.

If we know those two positions in the game world -- the camera position and the pixel position -- then finding out what's under that pixel is a trivial problem.

Now, we already know where the camera is in our game world, so that part isn't a problem. The only tricky part is figuring out where in our game world that virtual screen pixel is.

To get that game-world screen pixel position, I do this (pseudocode follows):

// first, get the cursor's window position, in pixels.
Vector2D pixelPos2D = WindowMousePosition();

// convert pixelPos2D from the range  [(0,0) .. (width,height)] to 
// the range [(-1,-1) .. (1,1)]
pos2D.x = pos2D.x / (0.5f * WindowWidth()) - 1.0f;
pos2D.y = pos2D.y / (0.5f * WindowHeight()) - 1.0f;

// now work out how big our screen is, in world coordinates, 
// if it's one meter in front of the camera.
float vertFOV = CameraVerticalFieldOfView();
float halfVertFOV = 0.5f * vertFOV;
float screenHalfHeight = tanf(halfVertFOV);
float screenHalfWidth = screenHalfHeight * WindowAspectRatio();

// screenHalfHeight and screenHalfWidth tell us (half) the actual in-game
// size of our screen, assuming that it's placed one meter in front of 
// the camera.
//
// So relative to the camera, our pixel must be located at:
Vector3D pixelPos3D( pos2D.x * screenHalfWidth, pos2D.y * screenHalfHeight, 1.f );

// now, we need our pixel's position relative to the world, instead of
// relative to the camera, so we need to apply our camera's 3D 
// transformation to it.
Vector3D pixelWorldPos3D = camera->GetLocalToWorldMatrix() * pixelPos3D;
// (or however your maths library performs matrix multiplication)

(Note that in this code, I assume that our camera looks forward along its positive Z axis, and that its X and Y axes match your window's X and Y axes. If your camera looks in some other direction, or has swapped its X or Y directions compared against your window's coordinate system, you'll need to make minor modifications)

And that's pretty much it. Now you know where your camera is, now you know where that virtual "pixel" is, and so now you can fire a pretend 'bullet' along that line using whatever method you already use for collision detection. It will tell you exactly what you hit, and exactly where.

There are other ways to do this as well. As noted in the question's comments, gluUnProject() is a common approach. Folks often will recommend reading back a pixel of the z-buffer (generally from the previous frame) to determine the z-buffer depth at the pixel to be queried, and then feeding the 2D pixel position plus that depth value into gluUnProject(), you can get where that point was in world-space. Down-sides of this approach are that at the time of calling gluUnProject(), you need to know the modelview matrix, the projection matrix, and the viewport settings. Plus, of course, needing to read back a pixel of the z-buffer, which almost certainly will cause an OpenGL pipeline stall. Plus, you need to be very careful in your code execution; it's easy for your z-buffer to be a frame or two old, which can result in serious inaccuracies when the camera or objects in the world are moving quickly.

Alternately, as Lars points out, if you do know those various matrices and viewport settings, then you can call gluUnProject() with an arbitrary non-zero 'winZ' value, and it will give you the world position of the specified pixel, at that distance forward from the camera. Using a winZ value of '1' should give the same final position that is calculated by my pseudocode, above, so you could use that as a less-code-required way to generate the direction for your virtual 'bullet' to fire, if you know the other information which it needs.