What could cause a pixel shader to paint outside the lines of the vertex shader output?

Question

From what I understand, the pixels that a pixel shader operates on are specified implicitly by the SV_POSITION output (in DirectX) of the vertex shader. What then could cause a pixel shader to render in the middle of nowhere?

I used the new Visual Studio 2012 graphics debugger to visualize my vertex and pixel shader output. This is the output from a DrawIndexed() call that draws a cube:

The pink part is the rendered output of the pixel shader, which takes the cube on its left as its input.

The vertex shader code:

cbuffer Buf { float4x4 final; };

struct In
{
    float4 pos:POSITION;
    float3 norm:NORMAL;
    float2 texuv:TEXCOORD;
};

struct Out
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

Out main(In input)
{
    Out output;
    output.pos = mul(input.pos, final);
    output.col = float4(1.0f, 0.5f, 0.5f, 1.0f);
    output.tex = input.texuv;

    return output;
}

And the pixel shader:

struct In
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

float4 main(In input) : SV_TARGET
{
    return input.col;
}

The raster stage is the only thing between the vertex shader and the pixel shader, so my suspicion is that it's some raster stage settings. But the raster stage shouldn't change the shape of the vertex shader output so drastically, should it?

Answer 1

I just had a look at my own vertex shaders, the difference is likely in the matrix you use.

If you are working with the transpose of the matrix, vector*matrix is appropriate, else you will need to do matrix*vector.

PS_IN VS(VS_IN input)
{
    PS_IN output;
    float4x4 WVP = transpose(VP);
    output.pos = mul(input.pos, WVP);
    return output;
}

Does exactly the same as

PS_IN VS(VS_IN input)
{
    PS_IN output;
    output.pos = mul(VP, input.pos);  
    return output;
}

Answer 2

Assuming the debug images above are correct, there is either

a) something wrong with the way the images you are debugging are being selected/updated, or b) The debug image of the vertex shader is not showing the transformations being applied, for whatever reason. As a rule, whatever comes out of the vertex shader should come into the pixel shader as a filled triangle with the exact coordinates specified by the vertex shader. c) You are not building with the correct settings for debugging/performance testing

Easiest fix: use a working project (such as one of the DirectX Samples), and get that working properly, then isolate the problem if you can.