Game Editor - When screen is clicked, how do you identify which object that is clicked?

I'm trying to create a Game Editor, currently just placing different types of Shapes and such. I'm doing this in Windows Forms while drawing the 3D with XNA. So, if I have a couple of Shapes on the screen and I click the screen I want to be able to identify "which" of these objects you clicked.

What is the best method for this? Since having two objects one behind the other, it should be able to recognize the one in front and not the one behind it and also if I rotate the camera and click on the one behind it - it should identify it and not the first one.

• The process of determining which object has been clicked on is usually referred to as 'picking'. Some Google searching for something like 'xna picking' should give you some useful results. Jun 30 '12 at 11:24

In this video I show how do it with code...the projection/view is isometric, but all calcs are done in 3D...

When I calculate the mouse ray, I check for collision against a plane... but the procedure is analog to check against a list of boundingspheres associated with your objects.

The algorithm should be similar to this:

1. Calculate mouse ray
2. for every object
3.     check if ray intersecs with it
4.         if does then save the object and the distance in a sorted list by distance
5. choose from sorted list the nearest object.

• Your example was pretty much what I'm looking for, perhaps not Isometric like. How do you render that Grid and that Outline of a Cube? Need something like that to test where the mouse is located in 3D Space. Jun 30 '12 at 18:50
• I use an own class to draw lines... is easy to implement through this sentence GraphicsDevice.DrawUserPrimitives<VertexPositionColor> (PrimitiveType.LineList, data.Vertices, 0, data.Index / 2);
– Blau
Jun 30 '12 at 18:55
• Yes, I tried something similiar but I can't get it to work. I use a mathematical method for this, but my understanding of the method isn't what I think it is. My question about this: gamedev.stackexchange.com/questions/31424/… Jun 30 '12 at 19:25
• Here is the code to draw it pastebin.com/YpCKAHcF is easy... only draw vertical and horizontal lines
– Blau
Jun 30 '12 at 19:31
• You call DrawLine3D, but there's no code of it. Is it just DrawUserPrimitives(LineList, Data.Vertices, 0, data.Index/2)? data.Index / 2 is what? Jun 30 '12 at 19:46

The first thing you need to have a good grasp of is transforming 3D points to 2D screen coordinates and back again. Here are a couple of answers I gave about that subject; Understanding 3D to Screen, 3D to 2D.

Now, to go from your mouse coordinate in 2D to 3D you need to do a 3D to 2D transformation backwards. This is fairly simple.

An object/local coordinate is transformed to screen space by multiplying it with the world matrix (get the world coordinate), then by the view matrix (make the world coordinate relative to the camera) and then the projection matrix (get the screen coordinate).

Doing this backwards is easy. Invert all the matrices. Then multiply by the inverted projection, then the inverted view matrix. Now, you won't have a 3D coordinate for your mouse. The reason is because you were missing a dimension of data. So what you have is a ray instead. But that's good. The ray will have a defining coordinate and will be pointed in the same direction as the camera look at vector. You may also notice you don't need to invert the world matrix for any object. That's just because you don't really want the ray in object/local space, there's no point, you probably have your collision volumes defined in world space too.

Now all you have to do is circle/box/poly ray intersection testing to work out if the ray intersects with an object.

Find all of the objects that are intersected and keep a list. Then work out the distance of each object from the camera. The one closest to the camera is the one the user wanted to pick.

Here's the code you would need to do this in XNA. This solution should also work for other kinds of projects (as long as you convert it first). If you are only rendering 2D sprites then you're probably not using a projection matrix, so just hand the function the Matrix.Identity.

using System;
using Microsoft.Xna.Framework;

namespace FluxPrototype
{
/// <summary>
/// Contains functions useful in working with XNA Vectors.
/// </summary>
static class VectorHelper
{
/// <summary>
/// Converts a Vector2 position into a 3D ray.
/// </summary>
/// <param name="Position">The 2D position.</param>
/// <param name="View">The view matrix for the camera.</param>
/// <param name="Projection">The projection matrix for the camera.</param>
/// <param name="Point">The returned point defining part of the 3D ray.</param>
/// <param name="Direction">The direction of the 3D ray.</param>
public static void Unproject(Vector2 Position, Matrix View, Matrix Projection, out Vector3 Point, out Vector3 Direction)
{
if (Position == null)
Position = Vector2.Zero;

// Create two 3D points from the position. The first one we will return.
Point = new Vector3(Position, 0);
Vector3 Point2 = new Vector3(Position, 1);

// Transform the points.
Matrix InvertedProjection = Matrix.Invert(Projection);
Point = Vector3.Transform(Point, InvertedProjection);
Point2 = Vector3.Transform(Point2, InvertedProjection);

Matrix InvertedView = Matrix.Invert(View);
Point = Vector3.Transform(Point, InvertedView);
Point2 = Vector3.Transform(Point2, InvertedView);

// Use the difference between the points to define the ray direction.
// This will only be the camera direction if the view frustum is orthographic.
Direction = (Point2 - Point);
Direction.Normalize();
}
}
}

• Ok. I'll look into it. Care to explain how to invert Matrices? I have no idea how. Jun 29 '12 at 14:00
• Matrix InvertedView = Matrix.Invert(View); Jun 29 '12 at 15:35
• Does this method work when placing stuff on the Editor? Since that's more "important" than selecting them. Say if I want to add a cube to the screen I click where I want it - does this method support that too? Jun 30 '12 at 13:08
• This is as far as I've come and I'm not sure if it's right: Vector2 screenPosition = new Vector2(PointToClient(MousePosition).X, PointToClient(MousePosition).Y); Matrix m = screenPosition * matrix.Invert(Camera.MatrixWorld) * Matrix.Invert(Camera.MatrixView) * Matrix.Invert(Camera.MatrixWorld); Jun 30 '12 at 13:18
• ...and I create my Matrices (in my Camera class) this way: matrix_world = Matrix.CreateRotationZ(MathHelper.ToRadians(world_rotation.Z)) * Matrix.CreateRotationY(MathHelper.ToRadians(world_rotation.Y)) * Matrix.CreateRotationX(MathHelper.ToRadians(world_rotation.X)) * Matrix.CreateTranslation(world_location) * Matrix.CreateScale(world_zoom); matrix_view = Matrix.CreateLookAt(view_position, view_lookat, Vector3.Up); matrix_projection = Matrix.CreatePerspectiveFieldOfView(projection_fieldofview, projection_aspectratio, projection_clipstart, projection_clipend); Jun 30 '12 at 13:18

Typically, I'll just create a bounding volume out of my mouse. The origin being where I clicked. Then, complete an intersection test by looping all objects. Then, sort the list you get back based on the criteria you want and select element 0.

• I get that. But how do I do with Z value? Or depth, because if I just click on the screen I get X, Y coordinate and that can later correspond to a point in the 3D World where I click, but the depth value I don't get. How "close" to the camera is it or how far away? Jun 29 '12 at 13:59

An idea I have considered but not yet tried:

Draw an offscreen image the same as the real image but no textures etc. Simply draw your objects, each object in a different "color" where the "color" is really just an object number. To figure out what's under the mouse you simply look at the color of the pixel on the alternate image.

Of course this is doing a lot of wasted work but it's the GPU doing it rather than the CPU and it's sure to resolve things by exactly the same rules.