It seems clear that Unity UI has some use for shader stencils, based on the fact that UI shaders tend to have something like this:

    Properties {
      [PerRendererData] _MainTex ("Font Texture", 2D) = "white" {}
      _Color ("Tint", Color) = (1,1,1,1)
      _StencilComp ("Stencil Comparison", Float) = 8
      _Stencil ("Stencil ID", Float) = 0
      _StencilOp ("Stencil Operation", Float) = 0
      _StencilWriteMask ("Stencil Write Mask", Float) = 255
      _StencilReadMask ("Stencil Read Mask", Float) = 255

All of those stencil properties end up being used in the ShaderLab Stencil command.

      Ref [_Stencil]
      Comp [_StencilComp]
      Pass [_StencilOp]
      ReadMask [_StencilReadMask]
      WriteMask [_StencilWriteMask]

Unfortunately this means that every aspect of whatever this shader is doing with the stencil is being controlled from somewhere else, which leaves it a mystery how exactly the stencil is being used. For example, is it true that the GUI.BeginGroup(Rect position) command uses the stencil to clip UI elements within the group from being drawn outside of the group? Does GUI.BeginScrollView(...) also work this way? If not, then what is the stencil being used for?

If one wanted to draw a mesh with GL commands or by some other means, and we wanted this mesh to be clipped so that it can be within a group or a scroll view within the Editor, then what sort of material could one use for rendering that mesh? Is it possible to setup the Stencil command in some correct way? Do the correct stencil property values change depending on when exactly the material is being rendered?

For example, one might imagine a static field on GUIUtility that might be called something like "stencilRef". Then BeginGroup could draw a filled rectangle on the stencil with some particular value, then set "stencilRef" to that value, and then any UI element could use that ref value to stencil itself to only being drawn within the bounds of the group. GUIUtility doesn't seem to have any such field, but does it actually exist somewhere else? Or does BeginGroup use some very different mechanism? And is that mechanism accessible for users to exploit?

Is it even possible for custom editor components to draw meshes in a way that respects clipping of BeginGroup or is this a power that is reserved only for the built-in tools provided by the Editor?

  • \$\begingroup\$ It sounds like your root question is actually "How to draw a mesh in a Unity editor window that is correctly clipped in groups or scroll views?" A full documentation of how stencils are used might not be necessary in order to get this one piece of functionality. \$\endgroup\$
    – DMGregory
    Mar 28, 2022 at 23:35
  • \$\begingroup\$ @DMGregory That does make sense, but ultimately the goal is a deeper understanding by whatever means. It has also occurred to me that we don't need to understand how the editor uses stencils when we can draw our own stencils to produce the same effect. But then I'll have learned nothing. \$\endgroup\$
    – Geo
    Mar 29, 2022 at 3:02


You must log in to answer this question.

Browse other questions tagged .