The first thing you need to have a good grasp of is transforming 3D points to 2D screen coordinates and back again. Here are a couple of answers I gave about that subject; Understanding 3D to Screen, 3D to 2D.
Now, to go from your mouse coordinate in 2D to 3D you need to do a 3D to 2D transformation backwards. This is fairly simple.
An object/local coordinate is transformed to screen space by multiplying it with the world matrix (get the world coordinate), then by the view matrix (make the world coordinate relative to the camera) and then the projection matrix (get the screen coordinate).
Doing this backwards is easy. Invert all the matrices. Then multiply by the inverted projection, then the inverted view matrix. Now, you won't have a 3D coordinate for your mouse. The reason is because you were missing a dimension of data. So what you have is a ray instead. But that's good. The ray will have a defining coordinate and will be pointed in the same direction as the camera look at vector. You may also notice you don't need to invert the world matrix for any object. That's just because you don't really want the ray in object/local space, there's no point, you probably have your collision volumes defined in world space too.
Now all you have to do is circle/box/poly ray intersection testing to work out if the ray intersects with an object.
Find all of the objects that are intersected and keep a list. Then work out the distance of each object from the camera. The one closest to the camera is the one the user wanted to pick.
Here's the code you would need to do this in XNA. This solution should also work for other kinds of projects (as long as you convert it first). If you are only rendering 2D sprites then you're probably not using a projection matrix, so just hand the function the Matrix.Identity.
using System;
using Microsoft.Xna.Framework;
namespace FluxPrototype
{
/// <summary>
/// Contains functions useful in working with XNA Vectors.
/// </summary>
static class VectorHelper
{
/// <summary>
/// Converts a Vector2 position into a 3D ray.
/// </summary>
/// <param name="Position">The 2D position.</param>
/// <param name="View">The view matrix for the camera.</param>
/// <param name="Projection">The projection matrix for the camera.</param>
/// <param name="Point">The returned point defining part of the 3D ray.</param>
/// <param name="Direction">The direction of the 3D ray.</param>
public static void Unproject(Vector2 Position, Matrix View, Matrix Projection, out Vector3 Point, out Vector3 Direction)
{
if (Position == null)
Position = Vector2.Zero;
// Create two 3D points from the position. The first one we will return.
Point = new Vector3(Position, 0);
Vector3 Point2 = new Vector3(Position, 1);
// Transform the points.
Matrix InvertedProjection = Matrix.Invert(Projection);
Point = Vector3.Transform(Point, InvertedProjection);
Point2 = Vector3.Transform(Point2, InvertedProjection);
Matrix InvertedView = Matrix.Invert(View);
Point = Vector3.Transform(Point, InvertedView);
Point2 = Vector3.Transform(Point2, InvertedView);
// Use the difference between the points to define the ray direction.
// This will only be the camera direction if the view frustum is orthographic.
Direction = (Point2 - Point);
Direction.Normalize();
}
}
}
