HLSL Pixel Shader for Depth Only Pass

Question

I'm using DirectX 11 and shader model 5.

There are many tutorials online for writing engines and shaders for shadow mapping. I have found an issue which seems to have a hard to find answer.

Typical shadow maps can be created by setting null render targets, assigning a depth/stencil, and setting a null pixel shader. What about the case where I want to sample a texture to determine an objects transparency and discard the pixel based on that transparency (i.e. shadow mapping a tree or chainlink fence)? In this case I need a pixel shader which samples a texture and outputs an alpha value to determine whether to write a pixel to the depth stencil. How do I output an alpha value if there are no render targets?

The regular pixel shader for an object might be:

struct OutputStruct
{
float4 Diffuse : SV_Target0;
float4 WorldPos : SV_Target1;
float4 Normal : SV_Target2;
}
OutputStruct PS(InputStruct Input)
{
    OutputStruct output = (OutputStruct)0;
    // perform pixel shading operations
    return output;
}

This seems to be what I want:

struct OutputStruct
{
}
OutputStruct PS(InputStruct Input)
{
    OutputStruct output = (OutputStruct)0;
    // perform pixel shading operations
    return output;
}

Where do I determine whether to write to the depth stencil based on the pixel shader operations? How do I output an alpha (0 or 1 in value)? Remember that the blend state is defined for each render target slot of which I have none.

Answer 1

The proper way of doing this is to implement the alpha test yourself in a pixel shader and clip (-1) any pixel that fails. This will discard the pixel and no depth or stencil values will be written to the framebuffer, without having to write anything to a separate color buffer and perform an extra test afterwards.

As discussed in comments, arbitrarily deciding during pixel shader execution to discard a pixel is going to disable some hardware depth buffer optimizations. However, the complexity of your depth-only shader is such that it probably will not affect performance too much. Early Z benefits you most when you can skip complicated pixel shaders (such as those used in lighting) for occluded primitives. This shader basically amounts to a single (probably cached) texture lookup and comparison between 0.0 per-pixel; not complicated at all.