How are Stencil Buffer Operations Applied in ShaderLab

Question

In Unity's Shader lab, I've read that the Stencil buffer is implemented with a mask, and an object reacting to the mask.

The mask may be defined like the following:

SubShader {
    Tags { 
        "RenderType"="Opaque" 
        "Queue" = "Geometry-100"
    }

    Stencil{
        Ref 1
        Comp always
        Pass replace
    }
    ...
}

To my understanding, because Comp is defined as always and Pass as replace, every pixel that this geometry covers, will have its stencil buffer replaced with 1.

The object to be masked may have a shader similar to the following:

SubShader
{
    Stencil{
        Ref 1
        Comp equal
        Pass keep
        Fail keep
    }
    ...
}

And supposedly, this object would only render if its is rendered over the pixels affected by the mask.

Does this mean that every object rendered with a shader that contains the Stencil function will automatically discard any pixels where the Comp fails, before passing it to the fragment shader?

Consider the following: There are no masks, but only a single object containing the Stencil function. If the stencil function is declared as

Stencil{
    Ref 1
    Comp never
}

then this object will never render to the screen? What is the default value in the stencil buffer? Does rendering path affect any of this?

Answer 1

I think the quirky part of stencil shaders is the overloading on the keywords:

• The reference value can be used for both conditional testing (e.g. in combination with pass greater) and for writing (pass replace)
• The comparison function can be used for controlling which pixels are drawn and/or for controlling which stencil values are written.

With that I'll try to answer your specific questions:

To my understanding, because Comp is defined as always and Pass as replace, every pixel that this geometry covers, will have its stencil buffer replaced with 1.

and

And supposedly, this object would only render if its is rendered over the pixels affected by the mask.

Are correct. I verified by reading Unity documentation and doing some tests in which I drew a sphere over a cube (mask) and only the area of intersection was drawn.

Does this mean that every object rendered with a shader that contains the Stencil function will automatically discard any pixels where the Comp fails, before passing it to the fragment shader?

That is true, however the shader could still write to the stencil buffer - for example Fail IncrSat would increment the value in the stencil buffer. So objects with Comp never will not render to the screen. An exception would be if you have shadow map shadows enabled, then shadows can still be drawn (where the object is, and shadows casted onto other objects) even though your pixels are not - that is because shadows are drawn in a separate pass.

What is the default value in the stencil buffer?

The default value is zero. Unity cameras allow you to select what is cleared (depth, color, none etc) however there are no options to control if the stencil buffer is cleared and to what value. If you check in the frame buffer tool, then you'll be able to see exactly where the stencil buffer is cleared - it will show up as Clear (color+z+stencil).

Does rendering path path affect any of this?

Generally, stencils should work in both deferred and forward. In deferred mode you may want to use this function https://docs.unity3d.com/ScriptReference/Camera-clearStencilAfterLightingPass.html to clear the buffer. There are also some limitations in deferred mode. See the deferred lighting path section on this page:

Stencil functionality for objects rendered in the deferred rendering path is somewhat limited, as during the base pass and lighting pass the stencil buffer is used for other purposes. During those two stages stencil state defined in the shader will be ignored and only taken into account during the final pass. Because of that it’s not possible to mask out these objects based on a stencil test, but they can still modify the buffer contents, to be used by objects rendered later in the frame. Objects rendered in the forward rendering path following the deferred path (e.g. transparent objects or objects without a surface shader) will set their stencil state normally again.

The deferred rendering path uses the three highest bits of the stencil buffer, plus up to four more highest bits - depending on how many light mask layers are used in the scene. It is possible to operate within the range of the “clean” bits using the stencil read and write masks, or you can force the camera to clean the stencil buffer after the lighting pass using Camera.clearStencilAfterLightingPass.