1
\$\begingroup\$

I never used the stencil buffer for anything until now, but I want to change this. I have an idea of how it should work: the gpu discards or keeps rasterized pixels before the pixel shader based on the stencil buffer value on the given position and some stencil operation.

What I don't know is how would I mark a pixel in the stencil buffer with a specific value. For example I draw my scene and want to mark everything which is drawn with a specific material (this material could be looked up from a texture so ideally I should mark the pixel in the pixel shader), so that later when I do some post processing on my scene I would only do it on the marked pixels. I didn't find anything on the internet besides how to set up a stencil buffer and explaining the different stencil operations. I was expecting to find some System-Value semantics like SV_Depth to write to in the pixel shader (because the stencil buffer shares the same resource with the depth buffer in D3D11), but there is no such thing on MSDN. So how should I do this?
If I am misunderstanding something please help me clear that up.

\$\endgroup\$
  • 1
    \$\begingroup\$ take a look at stencil chapter in open.gl it's obviously opengl tutorial, but principles are the same \$\endgroup\$ – Ali1S232 Jun 8 '14 at 3:56
  • 1
    \$\begingroup\$ Since you cannot actually write an explicit value to the stencil buffer from a shader, it sounds like what you need to do is test the material in a pixel shader and clip it if it does not match. Clipped fragments will not write to the stencil buffer. That will create a stencil mask that does what you want, but it will basically require an additional pass to build the mask. \$\endgroup\$ – Andon M. Coleman Jun 8 '14 at 4:20
  • \$\begingroup\$ I see so I can't actually mark them in the shader only through changing the depth stencil states before draw calls, I find that somewhat disappointing. Feel free to write this as an answer if you want. @AndonM.Coleman \$\endgroup\$ – János Turánszki Jun 8 '14 at 4:33
3
\$\begingroup\$

There are two ways to set the stencil buffer value in Direct3D 11, one of which is only available in Direct3D 11.3 (and Direct3D 12). I will split this answer into two parts accordingly.

Direct3D 11 General

As part of the D3D11_DEPTH_STENCIL_DESC you specify what action to take on a stencil test pass and fail. These options boil down to:

  • Keep the current value in the stencil buffer (D3D11_STENCIL_OP_KEEP)
  • Set the current value in the stencil buffer to zero (D3D11_STENCIL_OP_ZERO)
  • Increment the current value in the stencil buffer (D3D11_STENCIL_OP_INCR/_SAT)
  • Decrement the current value in the stencil buffer (D3D11_STENCIL_OP_DECR/_SAT)
  • Replace the current value in the stencil buffer with the reference value (D3D11_STENCIL_OP_REPLACE)

That last option is how you can explicitly set a stencil buffer index to a certain value.

The reference value is passed as the second parameter of the OMSetDepthStencilState method. Keep in mind that while the reference value argument takes in a UINT (32-bit), depth-stencil buffers typically allot only 8 bits to the stencil portion (DXGI_FORMAT_D24_UNORM_S8_UINT).

Direct3D 11.3 and Direct3D 12

In Direct3D 11.3, a new shader semantic called SV_StencilRef was introduced as a target for pixel shader output. With this you can explicitly set the stencil buffer value on a per-pixel basis (as opposed to changing the stencil reference value on a per-call basis).

Support for this feature can be queried via:

D3D11_FEATURE_DATA_D3D11_OPTIONS2 featuresDescr;
ZeroMemory(&featuresDescr, sizeof(D3D11_FEATURE_DATA_D3D11_OPTIONS2));

d3d11Device->CheckFeatureSupport(D3D11_FEATURE_D3D11_OPTIONS2, &featuresDescr, sizeof(D3D11_FEATURE_DATA_D3D11_OPTIONS2));

if(featuresDescr.PSSpecifiedStencilRefSupported)
{
    // SV_StencilRef is supported
}
\$\endgroup\$
  • \$\begingroup\$ This question is rather old, but still turns up near the top of Google searches about stencil buffers. So figured I might give it an actual answer. \$\endgroup\$ – ssell Jun 20 '16 at 5:13

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.