1
\$\begingroup\$

From what I understand, the pixels that a pixel shader operates on are specified implicitly by the SV_POSITION output (in DirectX) of the vertex shader. What then could cause a pixel shader to render in the middle of nowhere?

I used the new Visual Studio 2012 graphics debugger to visualize my vertex and pixel shader output. This is the output from a DrawIndexed() call that draws a cube:

enter image description here

The pink part is the rendered output of the pixel shader, which takes the cube on its left as its input.

The vertex shader code:

cbuffer Buf { float4x4 final; };

struct In
{
    float4 pos:POSITION;
    float3 norm:NORMAL;
    float2 texuv:TEXCOORD;
};

struct Out
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

Out main(In input)
{
    Out output;
    output.pos = mul(input.pos, final);
    output.col = float4(1.0f, 0.5f, 0.5f, 1.0f);
    output.tex = input.texuv;

    return output;
}

And the pixel shader:

struct In
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

float4 main(In input) : SV_TARGET
{
    return input.col;
}

The raster stage is the only thing between the vertex shader and the pixel shader, so my suspicion is that it's some raster stage settings. But the raster stage shouldn't change the shape of the vertex shader output so drastically, should it?

\$\endgroup\$
  • \$\begingroup\$ Just out of curiosity, why are you doing mul(input.pos, final)? Isn't it usually Matrix-Vector multiplication, instead of vector-matrix? or does HLSL not care about the order? \$\endgroup\$ – user13213 Jun 19 '12 at 10:57
  • 1
    \$\begingroup\$ @melak47 Hah, what do you know -- that turns out to be the cause! No idea why the vertex shader output is shown to be correct though... \$\endgroup\$ – Rei Miyasaka Jun 19 '12 at 12:12
  • \$\begingroup\$ @melak47 -- Do you mind making that an answer so I can mark this question answered? What's really odd though is that I see it written vector-matrix a lot, and it seems to work fine for them.... \$\endgroup\$ – Rei Miyasaka Jun 19 '12 at 12:15
  • 1
    \$\begingroup\$ Thank you, by the way. I had no idea the debugging they integrated in VS2012 was actually for D3D10 and up, I've always had trouble with PIX, but this works perfectly now! \$\endgroup\$ – user13213 Jun 19 '12 at 13:18
2
\$\begingroup\$

I just had a look at my own vertex shaders, the difference is likely in the matrix you use.

If you are working with the transpose of the matrix, vector*matrix is appropriate, else you will need to do matrix*vector.

PS_IN VS(VS_IN input)
{
    PS_IN output;
    float4x4 WVP = transpose(VP);
    output.pos = mul(input.pos, WVP);
    return output;
}

Does exactly the same as

PS_IN VS(VS_IN input)
{
    PS_IN output;
    output.pos = mul(VP, input.pos);  
    return output;
}
\$\endgroup\$
  • \$\begingroup\$ Thanks, and glad you got something useful out of the question too! \$\endgroup\$ – Rei Miyasaka Jun 19 '12 at 20:32
0
\$\begingroup\$

Assuming the debug images above are correct, there is either

a) something wrong with the way the images you are debugging are being selected/updated, or b) The debug image of the vertex shader is not showing the transformations being applied, for whatever reason. As a rule, whatever comes out of the vertex shader should come into the pixel shader as a filled triangle with the exact coordinates specified by the vertex shader. c) You are not building with the correct settings for debugging/performance testing

Easiest fix: use a working project (such as one of the DirectX Samples), and get that working properly, then isolate the problem if you can.

\$\endgroup\$
  • \$\begingroup\$ The vertex and pixel shader stages in the debugger show up fine with other code, but I'm assuming that's because it's also rendering fine. I thought it might be the debugger too, but sure enough the output on the screen with my code is exactly what's showing in the pixel shader output in the debugger, and what's showing up in the vertex shader is the correct cube. \$\endgroup\$ – Rei Miyasaka Jun 19 '12 at 5:32

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.