How to convert screen space into 3D world space?

I am trying to make a 3D model to look at the mouse position. Currently I have this code:

MouseState mouseState = Mouse.GetState();

Matrix world = Matrix.CreateTranslation(0, 0, 0);

Vector3 source = new Vector3((float) mouseState.X, 1f, (float) mouseState.Y);
Vector3 mousePoint = GraphicsDevice.Viewport.Unproject(source, this.game.Camera.Projection, this.game.Camera.View, world);

System.Console.Out.WriteLine("x: " + mousePoint.X + ", Y: " + mousePoint.Y + ", Z: " + mousePoint.Z);


I get output such as this in the console:

x: 0.1787011, Y: 999.7204, Z: 299.7723
x: 0.08917224, Y: 999.8602, Z: 299.8862
x: 0.05908892, Y: 999.9069, Z: 299.9241
x: 0.04422692, Y: 999.9302, Z: 299.9431
x: 0.03303542, Y: 999.9477, Z: 299.9573
x: 0.03109217, Y: 999.9508, Z: 299.9598
x: 0.02930491, Y: 999.9535, Z: 299.9621
x: 0.02770578, Y: 999.9559, Z: 299.9641
x: 0.0263205, Y: 999.9581, Z: 299.9659


The player is positioned at Vector3.Zero. The camera is positioned at new Vector3(0.0f, 1000.0f, 300.0f).

I would like to get output such as X: -300, Y: 0, Z: -300 when moving the mouse to screen top left and when moving the mouse to screen center where the model is located I want to have X: 0, Y: 0, Z: 0. So I want the world coordinates of the mouse so that I can make the player rotate towards the mouse.

Basically I want to map mouse X into world X, mouse Y to world Z and the world Y coordinate can be 0 at all times.

• keep in mind that when add a dimension to a 2d point, it becomes a 3d line that is infinitely kind. There is no such thing as converting a 2 d point to a 3d point without more input, usually a world space distance from the camera. – Steve H Feb 5 '12 at 18:54

You need the inverse of the camera_to_screen matrix.

The 3D pipeline overall looks like this:

model -> world -> camera -> viewport -> screen


The world_to_camera matrix is the usual rotation/translation matrix you construct for a 3D camera system. Inverting it is trivial: https://stackoverflow.com/questions/695043/how-does-one-convert-world-coordinates-to-camera-coordinates

The camera->viewport matrix is your projection matrix; inverting that is trivial if you know how to construct it in the first place: http://www.gamedev.net/topic/478055-perspective-projection-matrix/

The viewport_to_screen matrix is generally a simple transformation that maps the coordinate space [-1,-1]:[1,1] into [0,0];[ScreenWidth,ScreenHeight]. You don't usually need to generate this matrix yourself as it's taken care of by the graphics hardware automatically. The order of operations of the matrix is to scale by half the screen size, then to translate by half the screen size. Inverting those operations is basic algebra.

Multiply those inverted matrices together in the opposite order to get your screen->world matrix. You can now multiply by the vector (ScreenX,ScreenY,Near,1) to get the world position of the mouse cursor on the near clipping plane in world space.

If you need to do object picking at some point, you can take that point and the camera's position to generate a ray to do ray-collision test to find the object the cursor is pointing at.

• I have revised my question, do you know what's the problem with it? – rFactor Feb 5 '12 at 15:16

Xna has a built im method for doing this. GraphicsDevice.Viewport.Unproject

Checkout the code sample from here to see how to implement it:

http://create.msdn.com/en-US/education/catalog/sample/picking_triangle

• The problem is that there is nothing I am picking. The world does not even have a terrain yet. All I want is to know the X and Z coordinates the mouse is at on the map, in order to rotate a model towards the mouse. – rFactor Feb 5 '12 at 14:22
• I have revised my question, do you know what's the problem with it? – rFactor Feb 5 '12 at 15:16
• The sample demonstrates how to do a screenspace to 3d world space conversion. You can use the conversion for whatever you WA t to. In the samples case, they used it to help pick, that doesn't mean you can't use the conversion result for whatever you want. – Steve H Feb 5 '12 at 17:53
• To rotate a model based on mouse location or mouse movement, it is not necessary to convert screenspace to 3d world space. A change in rotation can be defined by an axis and angle. There is already a correlation from screen space to world space that you can use. It is the basis vectors of the view matrix. Would you lime to learn more about that approach? – Steve H Feb 5 '12 at 18:06

I solved it after days of googling. I had to create a plane to ray cast to, then calculate denominator and numerator.

        /// <summary>
/// Returns the mouse X, Y and Z coordinates on the world so that the Y is always on ground zero (0).
/// </summary>
/// <param name="game"></param>
/// <returns></returns>
public static Vector3 GetMouseWorldPosition(Game game)
{
GraphicsDevice graphicsDevice = game.GraphicsDevice;
MouseState mouseState = Mouse.GetState();

Vector3 nearSource = new Vector3((float) mouseState.X, (float) mouseState.Y, 0f);
Vector3 farSource = new Vector3((float) mouseState.X, (float) mouseState.Y, 1f);
Vector3 nearPoint = graphicsDevice.Viewport.Unproject(nearSource, game.Camera.Projection, game.Camera.View, Matrix.Identity);
Vector3 farPoint = graphicsDevice.Viewport.Unproject(farSource, game.Camera.Projection, game.Camera.View, Matrix.Identity);

// Create a ray from the near clip plane to the far clip plane.
Vector3 direction = farPoint - nearPoint;
direction.Normalize();

// Create a ray.
Ray ray = new Ray(nearPoint, direction);

// Calculate the ray-plane intersection point.
Vector3 n = new Vector3(0f, 1f, 0f);
Plane p = new Plane(n, 0f);

// Calculate distance of intersection point from r.origin.
float denominator = Vector3.Dot(p.Normal, ray.Direction);
float numerator = Vector3.Dot(p.Normal, ray.Position) + p.D;
float t = -(numerator / denominator);

// Calculate the picked position on the y = 0 plane.
Vector3 pickedPosition = nearPoint + direction * t;

return pickedPosition;
}

• When you get a chance, and are able to, please mark this as the correct answer, thanks. – John McDonald Feb 5 '12 at 20:54
• Yes, that will be tomorrow because SO won't allow me to mark it yet. – rFactor Feb 5 '12 at 21:37