Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

From what I understand, the pixels that a pixel shader operates on are specified implicitly by the SV_POSITION output (in DirectX) of the vertex shader. What then could cause a pixel shader to render in the middle of nowhere?

I used the new Visual Studio 2012 graphics debugger to visualize my vertex and pixel shader output. This is the output from a DrawIndexed() call that draws a cube:

enter image description here

The pink part is the rendered output of the pixel shader, which takes the cube on its left as its input.

The vertex shader code:

cbuffer Buf { float4x4 final; };

struct In
{
    float4 pos:POSITION;
    float3 norm:NORMAL;
    float2 texuv:TEXCOORD;
};

struct Out
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

Out main(In input)
{
    Out output;
    output.pos = mul(input.pos, final);
    output.col = float4(1.0f, 0.5f, 0.5f, 1.0f);
    output.tex = input.texuv;

    return output;
}

And the pixel shader:

struct In
{
    float4 col:COLOR;
    float2 tex:TEXCOORD;
    float4 pos:SV_POSITION;
};

float4 main(In input) : SV_TARGET
{
    return input.col;
}

The raster stage is the only thing between the vertex shader and the pixel shader, so my suspicion is that it's some raster stage settings. But the raster stage shouldn't change the shape of the vertex shader output so drastically, should it?

share|improve this question
    
Just out of curiosity, why are you doing mul(input.pos, final)? Isn't it usually Matrix-Vector multiplication, instead of vector-matrix? or does HLSL not care about the order? –  melak47 Jun 19 '12 at 10:57
1  
@melak47 Hah, what do you know -- that turns out to be the cause! No idea why the vertex shader output is shown to be correct though... –  Rei Miyasaka Jun 19 '12 at 12:12
    
@melak47 -- Do you mind making that an answer so I can mark this question answered? What's really odd though is that I see it written vector-matrix a lot, and it seems to work fine for them.... –  Rei Miyasaka Jun 19 '12 at 12:15
1  
Thank you, by the way. I had no idea the debugging they integrated in VS2012 was actually for D3D10 and up, I've always had trouble with PIX, but this works perfectly now! –  melak47 Jun 19 '12 at 13:18

2 Answers 2

up vote 2 down vote accepted

I just had a look at my own vertex shaders, the difference is likely in the matrix you use.

If you are working with the transpose of the matrix, vector*matrix is appropriate, else you will need to do matrix*vector.

PS_IN VS(VS_IN input)
{
    PS_IN output;
    float4x4 WVP = transpose(VP);
    output.pos = mul(input.pos, WVP);
    return output;
}

Does exactly the same as

PS_IN VS(VS_IN input)
{
    PS_IN output;
    output.pos = mul(VP, input.pos);  
    return output;
}
share|improve this answer
    
Thanks, and glad you got something useful out of the question too! –  Rei Miyasaka Jun 19 '12 at 20:32

Assuming the debug images above are correct, there is either

a) something wrong with the way the images you are debugging are being selected/updated, or b) The debug image of the vertex shader is not showing the transformations being applied, for whatever reason. As a rule, whatever comes out of the vertex shader should come into the pixel shader as a filled triangle with the exact coordinates specified by the vertex shader. c) You are not building with the correct settings for debugging/performance testing

Easiest fix: use a working project (such as one of the DirectX Samples), and get that working properly, then isolate the problem if you can.

share|improve this answer
    
The vertex and pixel shader stages in the debugger show up fine with other code, but I'm assuming that's because it's also rendering fine. I thought it might be the debugger too, but sure enough the output on the screen with my code is exactly what's showing in the pixel shader output in the debugger, and what's showing up in the vertex shader is the correct cube. –  Rei Miyasaka Jun 19 '12 at 5:32

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.