Unproject mouse position to tile -- extremely jumpy position?

I'm trying to unproject the mouse position to get the position on the X-Z plane of a ray cast from the mouse. The camera is fully controllable by the user. Right now, the algorithm I'm using is...

private bool getHoveredCorner(ArcBallCamera camera, System.Windows.Point m, out int tx, out int ty)
{
tx = -1; ty = -1;
Ray ray = camera.unproject((float) m.X, (float) m.Y);
Vector3 p = ray.Position + (-ray.Position.Y / ray.Direction.Y) * ray.Direction;
float u = p.X / totalWidth;
float v = p.Z / totalHeight;
if(u < 0 || u > 1 || v < 0 || v > 1) return false;
tx = (int) Math.Round(u * tileWidth);
ty = (int) Math.Round(v * tileHeight);
return tx >= 0 && tx <= tileWidth && ty >= 0 && ty <= tileHeight;
}

And camera.unproject is...

public Ray unproject(float x, float y)
{
Matrix m = viewProj;
Vector3 p1 = Vector3.Unproject(new Vector3(x, y, 0), 0, 0, width, height, nearPlane, farPlane, m);
Vector3 p2 = Vector3.Unproject(new Vector3(x, y, 1), 0, 0, width, height, nearPlane, farPlane, m);
Vector3 dir = p2 - p1;
dir.Normalize();
return new Ray(p1, dir);
}

The problem is that the projected position is "jumpy". As I make small adjustments to the mouse position, the projected point moves in strange ways. For example, if I move the mouse one pixel up, it will sometimes move the projected position down, but when I move it a second pixel, the project position will jump back to the mouse's location. The projected location is always close to where it should be, but it does not smoothly follow a moving mouse. The problem intensifies as I zoom the camera out. I'm not sure what's causing the problem, but I'm thinking it might be numerical instability?

EDIT 1: Video showing the problem

EDIT 2: A little snooping around in .NET Reflector on SlimDX.dll:

public static Vector3 Unproject(Vector3 vector, float x, float y, float width, float height, float minZ, float maxZ, Matrix worldViewProjection)
{
Vector3 coordinate = new Vector3();
Matrix result = new Matrix();
Matrix.Invert(ref worldViewProjection, out result);
coordinate.X = (float) ((((vector.X - x) / ((double) width)) * 2.0) - 1.0);
coordinate.Y = (float) -((((vector.Y - y) / ((double) height)) * 2.0) - 1.0);
coordinate.Z = (vector.Z - minZ) / (maxZ - minZ);
TransformCoordinate(ref coordinate, ref result, out coordinate);
return coordinate;
}

// ...

public static void TransformCoordinate(ref Vector3 coordinate, ref Matrix transformation, out Vector3 result)
{
Vector3 vector;
Vector4 vector2 = new Vector4 {
X = (((coordinate.Y * transformation.M21) + (coordinate.X * transformation.M11)) + (coordinate.Z * transformation.M31)) + transformation.M41,
Y = (((coordinate.Y * transformation.M22) + (coordinate.X * transformation.M12)) + (coordinate.Z * transformation.M32)) + transformation.M42,
Z = (((coordinate.Y * transformation.M23) + (coordinate.X * transformation.M13)) + (coordinate.Z * transformation.M33)) + transformation.M43
};
float num = (float) (1.0 / ((((transformation.M24 * coordinate.Y) + (transformation.M14 * coordinate.X)) + (coordinate.Z * transformation.M34)) + transformation.M44));
vector2.W = num;
vector.X = vector2.X * num;
vector.Y = vector2.Y * num;
vector.Z = vector2.Z * num;
result = vector;
}

...which seems to be a pretty standard method of unprojecting a point from a projection matrix, however this serves to introduce another point of possible instability. Still, I'd like to stick with the SlimDX Unproject routine rather than writing my own unless it's really necessary.

• Is your camera frustum plane orthogonally-aligned (that is no pitch, yaw, or roll) with respect to the tiles? Or is there arbitrary orientation / camera control in these respects? Dec 1 '12 at 8:36
• @Nick Wiggill - Arbitrary orientation; the user can control/rotate the camera. Dec 1 '12 at 8:50
• And to confirm, this jumping effect increases in severity from the centre of the viewport, out towards the edges? Or does it simply begin to occur with even severity at some point close to the edge? Dec 1 '12 at 8:55
• @Nick Wiggill -- Sorry, the question wasn't very clear. I've updated the question. Dec 1 '12 at 9:47
• You might want to try rendering both the snapped and unsnapped positions to help you see where it's going wrong. Also what are the near and far plane values that you're using?
Sep 12 '13 at 23:14