How to convert screen space into 3D world space?

Question

I am trying to make a 3D model to look at the mouse position. Currently I have this code:

MouseState mouseState = Mouse.GetState();

Matrix world = Matrix.CreateTranslation(0, 0, 0);

Vector3 source = new Vector3((float) mouseState.X, 1f, (float) mouseState.Y);
Vector3 mousePoint = GraphicsDevice.Viewport.Unproject(source, this.game.Camera.Projection, this.game.Camera.View, world);

System.Console.Out.WriteLine("x: " + mousePoint.X + ", Y: " + mousePoint.Y + ", Z: " + mousePoint.Z);

I get output such as this in the console:

x: 0.1787011, Y: 999.7204, Z: 299.7723
x: 0.08917224, Y: 999.8602, Z: 299.8862
x: 0.05908892, Y: 999.9069, Z: 299.9241
x: 0.04422692, Y: 999.9302, Z: 299.9431
x: 0.03303542, Y: 999.9477, Z: 299.9573
x: 0.03109217, Y: 999.9508, Z: 299.9598
x: 0.02930491, Y: 999.9535, Z: 299.9621
x: 0.02770578, Y: 999.9559, Z: 299.9641
x: 0.0263205, Y: 999.9581, Z: 299.9659

The player is positioned at Vector3.Zero. The camera is positioned at new Vector3(0.0f, 1000.0f, 300.0f).

I would like to get output such as X: -300, Y: 0, Z: -300 when moving the mouse to screen top left and when moving the mouse to screen center where the model is located I want to have X: 0, Y: 0, Z: 0. So I want the world coordinates of the mouse so that I can make the player rotate towards the mouse.

Basically I want to map mouse X into world X, mouse Y to world Z and the world Y coordinate can be 0 at all times.

Answer 1

You need the inverse of the camera_to_screen matrix.

The 3D pipeline overall looks like this:

model -> world -> camera -> viewport -> screen

The world_to_camera matrix is the usual rotation/translation matrix you construct for a 3D camera system. Inverting it is trivial: https://stackoverflow.com/questions/695043/how-does-one-convert-world-coordinates-to-camera-coordinates

The camera->viewport matrix is your projection matrix; inverting that is trivial if you know how to construct it in the first place: http://www.gamedev.net/topic/478055-perspective-projection-matrix/

The viewport_to_screen matrix is generally a simple transformation that maps the coordinate space [-1,-1]:[1,1] into [0,0];[ScreenWidth,ScreenHeight]. You don't usually need to generate this matrix yourself as it's taken care of by the graphics hardware automatically. The order of operations of the matrix is to scale by half the screen size, then to translate by half the screen size. Inverting those operations is basic algebra.

Multiply those inverted matrices together in the opposite order to get your screen->world matrix. You can now multiply by the vector (ScreenX,ScreenY,Near,1) to get the world position of the mouse cursor on the near clipping plane in world space.

If you need to do object picking at some point, you can take that point and the camera's position to generate a ray to do ray-collision test to find the object the cursor is pointing at.

Answer 2

Xna has a built im method for doing this. GraphicsDevice.Viewport.Unproject

Checkout the code sample from here to see how to implement it:

http://create.msdn.com/en-US/education/catalog/sample/picking_triangle

Answer 3

I solved it after days of googling. I had to create a plane to ray cast to, then calculate denominator and numerator.

        /// <summary>
        /// Returns the mouse X, Y and Z coordinates on the world so that the Y is always on ground zero (0).
        /// </summary>
        /// <param name="game"></param>
        /// <returns></returns>
        public static Vector3 GetMouseWorldPosition(Game game)
        {
            GraphicsDevice graphicsDevice = game.GraphicsDevice;
            MouseState mouseState = Mouse.GetState();

            Vector3 nearSource = new Vector3((float) mouseState.X, (float) mouseState.Y, 0f);
            Vector3 farSource = new Vector3((float) mouseState.X, (float) mouseState.Y, 1f);
            Vector3 nearPoint = graphicsDevice.Viewport.Unproject(nearSource, game.Camera.Projection, game.Camera.View, Matrix.Identity);
            Vector3 farPoint = graphicsDevice.Viewport.Unproject(farSource, game.Camera.Projection, game.Camera.View, Matrix.Identity);

            // Create a ray from the near clip plane to the far clip plane.
            Vector3 direction = farPoint - nearPoint;
            direction.Normalize();

            // Create a ray.
            Ray ray = new Ray(nearPoint, direction);

            // Calculate the ray-plane intersection point.
            Vector3 n = new Vector3(0f, 1f, 0f);
            Plane p = new Plane(n, 0f);

            // Calculate distance of intersection point from r.origin.
            float denominator = Vector3.Dot(p.Normal, ray.Direction);
            float numerator = Vector3.Dot(p.Normal, ray.Position) + p.D;
            float t = -(numerator / denominator);

            // Calculate the picked position on the y = 0 plane.
            Vector3 pickedPosition = nearPoint + direction * t;

            return pickedPosition;
        }