3
\$\begingroup\$

I'm using DirectX 11 and shader model 5.

There are many tutorials online for writing engines and shaders for shadow mapping. I have found an issue which seems to have a hard to find answer.

Typical shadow maps can be created by setting null render targets, assigning a depth/stencil, and setting a null pixel shader. What about the case where I want to sample a texture to determine an objects transparency and discard the pixel based on that transparency (i.e. shadow mapping a tree or chainlink fence)? In this case I need a pixel shader which samples a texture and outputs an alpha value to determine whether to write a pixel to the depth stencil. How do I output an alpha value if there are no render targets?

The regular pixel shader for an object might be:

struct OutputStruct
{
float4 Diffuse : SV_Target0;
float4 WorldPos : SV_Target1;
float4 Normal : SV_Target2;
}
OutputStruct PS(InputStruct Input)
{
    OutputStruct output = (OutputStruct)0;
    // perform pixel shading operations
    return output;
}

This seems to be what I want:

struct OutputStruct
{
}
OutputStruct PS(InputStruct Input)
{
    OutputStruct output = (OutputStruct)0;
    // perform pixel shading operations
    return output;
}

Where do I determine whether to write to the depth stencil based on the pixel shader operations? How do I output an alpha (0 or 1 in value)? Remember that the blend state is defined for each render target slot of which I have none.

\$\endgroup\$
  • 1
    \$\begingroup\$ I do not quite understand why you want to output an alpha value? You should be able to sample the texture and then clip (-1) directly from the pixel shader that writes depth for any alpha sample that fails; no color output necessary. You will make the shadow map generation perform slightly worse by doing this (it violates certain hardware optimizations to arbitrarily discard a pixel during the evaluation of a pixel shader), but it might not be all that bad considering how simple the pixel shader is. \$\endgroup\$ – Andon M. Coleman Mar 17 '15 at 20:17
  • \$\begingroup\$ "I do not quite understand why you want to output an alpha value?" Think of generating the shadow map which contains a tree. A quad has a texture on it which is transparent between the leaves. I, thus, do not want to write to the depth stencil when the alpha value of the tree texture is 0. Is this a case where I would use discard() or clip(-1)? \$\endgroup\$ – Russell Trahan Mar 17 '15 at 20:28
  • 2
    \$\begingroup\$ Yes, that is absolutely where you would use discard (GLSL) or clip(-1) (HLSL). It tells the fragment/pixel shader to throw away the results for the current pixel; this includes color, depth, stencil. It is as though the pixel never existed. In fact, in modern OpenGL there is no fixed-function alpha test anymore, you actually have to implement alpha testing yourself using discard. \$\endgroup\$ – Andon M. Coleman Mar 17 '15 at 20:59
  • \$\begingroup\$ OK. Put that as answer and I'll mark it. I was hesitant to use clip/discard because I have been under the impression that having that statement in a shader prevents early-z testing. \$\endgroup\$ – Russell Trahan Mar 17 '15 at 21:02
  • 1
    \$\begingroup\$ It does prevent early Z, which I pointed out in parentheses in my original comment. The thing is, a fixed-function alpha test also disables early Z. Any test that can only be performed after or during execution of a pixel/fragment shader disables early Z, and alpha testing (whether you implement it yourself or use some fixed-function feature) can only be done after color is computed per-pixel. \$\endgroup\$ – Andon M. Coleman Mar 17 '15 at 21:08
2
\$\begingroup\$

The proper way of doing this is to implement the alpha test yourself in a pixel shader and clip (-1) any pixel that fails. This will discard the pixel and no depth or stencil values will be written to the framebuffer, without having to write anything to a separate color buffer and perform an extra test afterwards.

As discussed in comments, arbitrarily deciding during pixel shader execution to discard a pixel is going to disable some hardware depth buffer optimizations. However, the complexity of your depth-only shader is such that it probably will not affect performance too much. Early Z benefits you most when you can skip complicated pixel shaders (such as those used in lighting) for occluded primitives. This shader basically amounts to a single (probably cached) texture lookup and comparison between 0.0 per-pixel; not complicated at all.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.