Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

Sign up
Here's how it works:
  1. Anybody can ask a question
  2. Anybody can answer
  3. The best answers are voted up and rise to the top

I'm having a bit of trouble with 3D picking, at first I thought my ray was inaccurate but it turns out that the picking is happening on faces facing the camera and faces facing away from the camera which I'm currently culling.

Here's my ray creation code, I'm pretty sure the problem isn't here but I've been wrong before.

    private uint Pick()
        Ray cursorRay = CalculateCursorRay();
        Vector3? point = Control.Mesh.RayCast(cursorRay);
        if (point != null)
            Tile hitTile = Control.TileMesh.GetTileAtPoint(point);
            return hitTile == null ? uint.MaxValue : (uint)(hitTile.X + hitTile.Y * Control.Generator.TilesWide);
        return uint.MaxValue;

    private Ray CalculateCursorRay()
        Vector3 nearPoint = Control.Camera.Unproject(new Vector3(Cursor.Position.X, Control.ClientRectangle.Height - Cursor.Position.Y, 0f));
        Vector3 farPoint = Control.Camera.Unproject(new Vector3(Cursor.Position.X, Control.ClientRectangle.Height - Cursor.Position.Y, 1f));
        Vector3 direction = farPoint - nearPoint;
        return new Ray(nearPoint, direction);

    public Vector3 Camera.Unproject(Vector3 source)
        Vector4 result;

        result.X = (source.X - _control.ClientRectangle.X) * 2 / _control.ClientRectangle.Width - 1;
        result.Y = (source.Y - _control.ClientRectangle.Y) * 2 / _control.ClientRectangle.Height - 1;
        result.Z = source.Z - 1;
        if (_farPlane - 1 == 0)
            result.Z = 0;
            result.Z = result.Z / (_farPlane - 1);
        result.W = 1f;
        result = Vector4.Transform(result, Matrix4.Invert(ProjectionMatrix));
        result = Vector4.Transform(result, Matrix4.Invert(ViewMatrix));
        result = Vector4.Transform(result, Matrix4.Invert(_world));
        result = Vector4.Divide(result, result.W);
        return new Vector3(result.X, result.Y, result.Z);

And my triangle intersection code. Ripped mainly from the XNA picking sample.

    public float? Intersects(Ray ray)
        float? closestHit = Bounds.Intersects(ray);
        if (closestHit != null && Vertices.Length == 3)
            Vector3 e1, e2;
            Vector3.Subtract(ref Vertices[1].Position, ref Vertices[0].Position, out e1);
            Vector3.Subtract(ref Vertices[2].Position, ref Vertices[0].Position, out e2);

            Vector3 directionCrossEdge2;
            Vector3.Cross(ref ray.Direction, ref e2, out directionCrossEdge2);
            float determinant;
            Vector3.Dot(ref e1, ref directionCrossEdge2, out determinant);
            if (determinant > -float.Epsilon && determinant < float.Epsilon)
                return null;
            float inverseDeterminant = 1.0f/determinant;
            Vector3 distanceVector;
            Vector3.Subtract(ref ray.Position, ref Vertices[0].Position, out distanceVector);

            float triangleU;
            Vector3.Dot(ref distanceVector, ref directionCrossEdge2, out triangleU);
            triangleU *= inverseDeterminant;
            if (triangleU < 0 || triangleU > 1)
                return null;
            Vector3 distanceCrossEdge1;
            Vector3.Cross(ref distanceVector, ref e1, out distanceCrossEdge1);
            float triangleV;
            Vector3.Dot(ref ray.Direction, ref distanceCrossEdge1, out triangleV);
            triangleV *= inverseDeterminant;
            if (triangleV < 0 || triangleU + triangleV > 1)
                return null;

            float rayDistance;
            Vector3.Dot(ref e2, ref distanceCrossEdge1, out rayDistance);
            rayDistance *= inverseDeterminant;
            if (rayDistance < 0)
                return null;
            return rayDistance;
        return closestHit;

I'll admit I don't fully understand all of the math behind the intersection and that is something I'm working on, but my understanding was that if rayDistance was less than 0 the face was facing away from the camera, and shouldn't be counted as a hit.

So my question is, is there an issue with my intersection or ray creation code, or is there another check I need to perform to tell if the face is facing away from the camera, and if so any hints on what that check might contain would be appreciated.

**Edit the working UnProject method.

public static Vector3 UnProject(ref Matrix4 projection, Matrix4 view, Size viewport, Vector3 mouse)
        Vector4 vec;

        vec.X = 2.0f * mouse.X / (float)viewport.Width - 1;
        vec.Y = -(2.0f * mouse.Y / (float)viewport.Height - 1);
        vec.Z = mouse.Z;
        vec.W = 1.0f;

        Matrix4 viewInv = Matrix4.Invert(view);
        Matrix4 projInv = Matrix4.Invert(projection);

        Vector4.Transform(ref vec, ref projInv, out vec);
        Vector4.Transform(ref vec, ref viewInv, out vec);

        if (vec.W > float.Epsilon || vec.W < float.Epsilon)
            vec.X /= vec.W;
            vec.Y /= vec.W;
            vec.Z /= vec.W;

        return vec.Xyz;
share|improve this question
May not related to the question, but your edited Unproject looks suspicious still: It is correct that you need to inverse the perspective divide in Unproject. But the safety check should prevent you from dividing by zero. Now it prevents you from dividing by eps? – Haymo Kutschbach Dec 25 '14 at 20:14
up vote 2 down vote accepted

This isn't a complete answer in itself, but this wouldn't fit into the comments below your question.

If believe this part of the Intersects function is what should be disallowing reverse facing triangles to be given as a result:

float triangleU;
Vector3.Dot(ref distanceVector, ref directionCrossEdge2, out triangleU);
triangleU *= inverseDeterminant;
if (triangleU < 0 || triangleU > 1)
    return null;

If this code was taken from the XNA sample, then you can be almost certain that the code from the sample is not part of the problem you're seeing, so either you have changed something in the sample code itself, or you're passing unexpected data into the methods from the sample. I can't pinpoint exactly where the problem is, but my best guess is that it lies somewhere in the Camera.Unproject function where the result is being modified.

share|improve this answer
Thanks for the tip, I'll look into un-projecting again and see if I can figure out if I'm doing something wrong there. If you see a problem with it please let me know. I'll also look over the Intersects method again to make sure I didn't mess up and copy something over incorrectly. – Tangeleno Apr 9 '12 at 6:11
I know it's been a while, but I just wanted to say thanks, It seems it was my Camera.UnProject method that was causing the issue, I've added my current UnProject to my main question incase it helps any one else, one odd thing however is I also needed to flip the nearPoint and farPoint vectors, not sure why exactly but it does work now. – Tangeleno Jun 24 '12 at 22:45

Your Answer


By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.