How to get depth of vertex in 0 - 1 range

Question

I want to get the depth of vertex in 0 to 1 range(like it done in the shader) and I do it like this

    D3DXVec3TransformCoord(&vertex, &vertex, &wvp);

depth = vertex.z;

It works(sometimes wrong) but when I write This

    if (depth > 1.0f || depth < 0.0f)return false;

Sometimes it returns false. How can I get correct depth for vertex in 0 to 1 range??

I want to compare two vertexes and select the nearest

EDIT

    wvp = worldmatrix*viewmatrix*projectionmatrix;

It works wrong when I zoom in

Answer 1

You are missing the normalization step, which requires a W component.

Hence, you should be calling D3DXVec4Transform and not D3DXVec3TransformCoord.

You need to feed this your vertex, with the last component (W) set to a 1.0 value.

The output vertex will have W set in the last component, and you need to divide by that:

v_in[3] = 1.0;

D3DXVec4Transform
(
    &v_out,
    &v_in,
    &wvp
);

v_out[0] = v_out[0] / v_out[3];
v_out[1] = v_out[1] / v_out[3];
v_out[2] = v_out[2] / v_out[3];

Now you can use v_out[2] as the depth value, but beware: it ranges from -1 to 1, and not from 0 to 1 as you assumed.

Answer 2

Matrix multiplication is not commutative. You need to do projection * view * model