From camera coordinates to world coordinates

Question

I want to calculate world coordinates from camera coordinates. However, I seem to have problems with my understandings of how matrices in HLSL work.

From world to camera is clear:

cameraPosition = mul(mul(worldPosition, view), projection);

Logic would now say that for the reverse, I could just use something like

worldPosition = mul(mul(cameraPosition, invProjection), invView);

However, when I check if it is correct with

cameraPosition = mul(mul(mul(mul(cameraPosition, invProjection), 
                                   invView), view), projection);

I don't get the same point back anymore.

The inverses should be fine as view * invView produces the identity matrix etc.

What is my misunderstanding here? Even the simpler case does not work:

void VS_test(in  float4 inPosition            : POSITION,
             out float4 outPosition           : POSITION)
{
    outPosition = inPosition;
}

produces the triangle I want. However, using

void VS_test(in  float4 inPosition            : POSITION,
             out float4 outPosition           : POSITION)
{
    outPosition = mul(mul(inPosition, view), invView);
}

already produces no visible triangle. Same with

void VS_test(in  float4 inPosition            : POSITION,
             out float4 outPosition           : POSITION)
{
    outPosition = mul(inPosition, mul(view, invView));
}

Pixel shader is just a shader which returns a constant color.

UPDATE

I have 3D camera-space coordinates WITH z-buffer value, like (0, 0, zNear) for the point directly in the center of the screen. I want to know what world coordinates correspond to this by doing the whole view-transform backwards.

Answer 1

I think that the problem is that you don't normalize screen space point after worldViewProj multiplication. This should do the job:

cameraPositionScreenSpace = mul(mul(worldPosition, view), projection);
cameraPositionScreenSpace /= cameraPositionScreenSpace .w;

Answer 2

Okay, stupid me. I accidently forwarded a wrong inverse matrix to the shader :-)

Answer 3

Since you mentioned the projection matrix in your code, I'm going to assume that you mean screen coordinates rather than camera coordinates.

While it's true that mathematically matrix * invMatrix results in an identity matrix, if one of these operations discards information (eg, turning 3D world coordinates into 2D screen coordinates), then you can't get that information back. You can cast rays out from the camera at a particular vector to try and hit something, but you can't reproduce the original coordinate.

Need proof?

(X * 0) / 0 == ? // Solve as an algorithm, not as algebra :)

Answer 4

I'm not entirely sure what you're trying to do but I'll give it my best shot =). First of all you are on the right track - to undo a transformation you must multiply the result by its inverse. So in a vertex shader you are taking an object/local space point and transforming it into homogeneous clip space for the pixel shader:

float4 outPos = mul(float4(inPos.xyz, 1), matWorldViewProj);

This can also be represented as:

float4 outPos = mul(mul(mul(float4(inPos.xyz, 1), matWorld), matView), matProj);

Thus to get the world space position of a point after a transformation into homogeneous clip space you must multiply the point by the inverse of the view and projection matrices:

float4 worldPos = mul(outPos, matInvViewProj);

Which can also be represented as:

float4 worldPos = mul(mul(outPos, matInvProj), matInvView);

So if you already have a vertex that's in view/eye (camera?) space, which means the origin of the coordinate system for that point is the current location of the view/eye:

float4 outPos = mul(float4(inPos.xyz, 1), matWorldView);

To get back the world position of that point you multiply it by the inverse of the view matrix:

float4 worldPos = mul(outPos, matInvView);