How can I mark a pixel in the stencil buffer?

Question

I never used the stencil buffer for anything until now, but I want to change this. I have an idea of how it should work: the gpu discards or keeps rasterized pixels before the pixel shader based on the stencil buffer value on the given position and some stencil operation.

What I don't know is how would I mark a pixel in the stencil buffer with a specific value. For example I draw my scene and want to mark everything which is drawn with a specific material (this material could be looked up from a texture so ideally I should mark the pixel in the pixel shader), so that later when I do some post processing on my scene I would only do it on the marked pixels. I didn't find anything on the internet besides how to set up a stencil buffer and explaining the different stencil operations. I was expecting to find some System-Value semantics like SV_Depth to write to in the pixel shader (because the stencil buffer shares the same resource with the depth buffer in D3D11), but there is no such thing on MSDN. So how should I do this?
If I am misunderstanding something please help me clear that up.

Answer 1

There are two ways to set the stencil buffer value in Direct3D 11, one of which is only available in Direct3D 11.3 (and Direct3D 12). I will split this answer into two parts accordingly.

Direct3D 11 General

As part of the D3D11_DEPTH_STENCIL_DESC you specify what action to take on a stencil test pass and fail. These options boil down to:

• Keep the current value in the stencil buffer (D3D11_STENCIL_OP_KEEP)
• Set the current value in the stencil buffer to zero (D3D11_STENCIL_OP_ZERO)
• Increment the current value in the stencil buffer (D3D11_STENCIL_OP_INCR/_SAT)
• Decrement the current value in the stencil buffer (D3D11_STENCIL_OP_DECR/_SAT)
• Replace the current value in the stencil buffer with the reference value (D3D11_STENCIL_OP_REPLACE)

That last option is how you can explicitly set a stencil buffer index to a certain value.

The reference value is passed as the second parameter of the OMSetDepthStencilState method. Keep in mind that while the reference value argument takes in a UINT (32-bit), depth-stencil buffers typically allot only 8 bits to the stencil portion (DXGI_FORMAT_D24_UNORM_S8_UINT).

Direct3D 11.3 and Direct3D 12

In Direct3D 11.3, a new shader semantic called SV_StencilRef was introduced as a target for pixel shader output. With this you can explicitly set the stencil buffer value on a per-pixel basis (as opposed to changing the stencil reference value on a per-call basis).

Support for this feature can be queried via:

D3D11_FEATURE_DATA_D3D11_OPTIONS2 featuresDescr;
ZeroMemory(&featuresDescr, sizeof(D3D11_FEATURE_DATA_D3D11_OPTIONS2));

d3d11Device->CheckFeatureSupport(D3D11_FEATURE_D3D11_OPTIONS2, &featuresDescr, sizeof(D3D11_FEATURE_DATA_D3D11_OPTIONS2));

if(featuresDescr.PSSpecifiedStencilRefSupported)
{
    // SV_StencilRef is supported
}