It seems clear that Unity UI has some use for shader stencils, based on the fact that UI shaders tend to have something like this:
Properties {
[PerRendererData] _MainTex ("Font Texture", 2D) = "white" {}
_Color ("Tint", Color) = (1,1,1,1)
_StencilComp ("Stencil Comparison", Float) = 8
_Stencil ("Stencil ID", Float) = 0
_StencilOp ("Stencil Operation", Float) = 0
_StencilWriteMask ("Stencil Write Mask", Float) = 255
_StencilReadMask ("Stencil Read Mask", Float) = 255
}
All of those stencil properties end up being used in the ShaderLab Stencil command.
Stencil
{
Ref [_Stencil]
Comp [_StencilComp]
Pass [_StencilOp]
ReadMask [_StencilReadMask]
WriteMask [_StencilWriteMask]
}
Unfortunately this means that every aspect of whatever this shader is doing with the stencil is being controlled from somewhere else, which leaves it a mystery how exactly the stencil is being used. For example, is it true that the GUI.BeginGroup(Rect position) command uses the stencil to clip UI elements within the group from being drawn outside of the group? Does GUI.BeginScrollView(...) also work this way? If not, then what is the stencil being used for?
If one wanted to draw a mesh with GL commands or by some other means, and we wanted this mesh to be clipped so that it can be within a group or a scroll view within the Editor, then what sort of material could one use for rendering that mesh? Is it possible to setup the Stencil command in some correct way? Do the correct stencil property values change depending on when exactly the material is being rendered?
For example, one might imagine a static field on GUIUtility that might be called something like "stencilRef". Then BeginGroup could draw a filled rectangle on the stencil with some particular value, then set "stencilRef" to that value, and then any UI element could use that ref value to stencil itself to only being drawn within the bounds of the group. GUIUtility doesn't seem to have any such field, but does it actually exist somewhere else? Or does BeginGroup use some very different mechanism? And is that mechanism accessible for users to exploit?
Is it even possible for custom editor components to draw meshes in a way that respects clipping of BeginGroup or is this a power that is reserved only for the built-in tools provided by the Editor?
