Unproject mouse position to tile -- extremely jumpy position?

Question

I'm trying to unproject the mouse position to get the position on the X-Z plane of a ray cast from the mouse. The camera is fully controllable by the user. Right now, the algorithm I'm using is...

    private bool getHoveredCorner(ArcBallCamera camera, System.Windows.Point m, out int tx, out int ty)
    {
        tx = -1; ty = -1;
        Ray ray = camera.unproject((float) m.X, (float) m.Y);
        Vector3 p = ray.Position + (-ray.Position.Y / ray.Direction.Y) * ray.Direction;
        float u = p.X / totalWidth;
        float v = p.Z / totalHeight;
        if(u < 0 || u > 1 || v < 0 || v > 1) return false;
        tx = (int) Math.Round(u * tileWidth);
        ty = (int) Math.Round(v * tileHeight);
        return tx >= 0 && tx <= tileWidth && ty >= 0 && ty <= tileHeight;
    }

And camera.unproject is...

    public Ray unproject(float x, float y)
    {
        Matrix m = viewProj;
        Vector3 p1 = Vector3.Unproject(new Vector3(x, y, 0), 0, 0, width, height, nearPlane, farPlane, m);
        Vector3 p2 = Vector3.Unproject(new Vector3(x, y, 1), 0, 0, width, height, nearPlane, farPlane, m);
        Vector3 dir = p2 - p1;
        dir.Normalize();
        return new Ray(p1, dir);
    }

The problem is that the projected position is "jumpy". As I make small adjustments to the mouse position, the projected point moves in strange ways. For example, if I move the mouse one pixel up, it will sometimes move the projected position down, but when I move it a second pixel, the project position will jump back to the mouse's location. The projected location is always close to where it should be, but it does not smoothly follow a moving mouse. The problem intensifies as I zoom the camera out. I'm not sure what's causing the problem, but I'm thinking it might be numerical instability?

EDIT 1: Video showing the problem

EDIT 2: A little snooping around in .NET Reflector on SlimDX.dll:

public static Vector3 Unproject(Vector3 vector, float x, float y, float width, float height, float minZ, float maxZ, Matrix worldViewProjection)
{
    Vector3 coordinate = new Vector3();
    Matrix result = new Matrix();
    Matrix.Invert(ref worldViewProjection, out result);
    coordinate.X = (float) ((((vector.X - x) / ((double) width)) * 2.0) - 1.0);
    coordinate.Y = (float) -((((vector.Y - y) / ((double) height)) * 2.0) - 1.0);
    coordinate.Z = (vector.Z - minZ) / (maxZ - minZ);
    TransformCoordinate(ref coordinate, ref result, out coordinate);
    return coordinate;
}

// ...

public static void TransformCoordinate(ref Vector3 coordinate, ref Matrix transformation, out Vector3 result)
{
    Vector3 vector;
    Vector4 vector2 = new Vector4 {
        X = (((coordinate.Y * transformation.M21) + (coordinate.X * transformation.M11)) + (coordinate.Z * transformation.M31)) + transformation.M41,
        Y = (((coordinate.Y * transformation.M22) + (coordinate.X * transformation.M12)) + (coordinate.Z * transformation.M32)) + transformation.M42,
        Z = (((coordinate.Y * transformation.M23) + (coordinate.X * transformation.M13)) + (coordinate.Z * transformation.M33)) + transformation.M43
    };
    float num = (float) (1.0 / ((((transformation.M24 * coordinate.Y) + (transformation.M14 * coordinate.X)) + (coordinate.Z * transformation.M34)) + transformation.M44));
    vector2.W = num;
    vector.X = vector2.X * num;
    vector.Y = vector2.Y * num;
    vector.Z = vector2.Z * num;
    result = vector;
}

...which seems to be a pretty standard method of unprojecting a point from a projection matrix, however this serves to introduce another point of possible instability. Still, I'd like to stick with the SlimDX Unproject routine rather than writing my own unless it's really necessary.

Answer 1

You might want to try rendering both the snapped to grid and unsnapped positions to help you see where it's going wrong.

You should also check that the near and far plane values that you're using aren't too far apart. Most notably the bigger far/near ratio is the less accuracy you'll get both for rendering and for the calculations.

Having said that the values of 1 and 500 you said you were using seem to be well within a reasonable range to give decent accuracy to me.

When you're looking for precision issues in floating point maths, the main thing to watch out for is subtraction. If the numbers you're subtracting are close together then you'll have less precision in the result. Have a read of at least the cancellation section of http://docs.oracle.com/cd/E19957-01/806-3568/ncg_goldberg.html for more details.

Answer 2

I found that I needed to tighten the near and far values and ensure that the unproject function's near/far plane input matches the projection matrix that is also fed into the unproject function. Remember, the projection matrix which helped form your 'viewProj' variable also contains near and far plane values.

For me, this resolved the exact symptoms you describe.