# How can I chain together Stencil tests from the same re-occurring routine, but keep the tests separate?

Sorry for the bad title; I don't know how else to explain this.

I have a class called a Container. An instance of Container has other elements "inside" it. These elements have independent positions that are relative to the position of the container.

So if I move the container, the elements inside of it are moved along with it; but keep their distance relative to each other.

This Container is therefore like a window through which you can view it's contents.

So, I would like to use a Stencil buffer for this. I have the following code:

glEnable(GL_STENCIL_TEST);
glClear(GL_STENCIL_BUFFER_BIT);
glStencilMask(0xFF);    // everything is written to the buffer
glStencilFunc(GL_ALWAYS, 1, 0xFF);
glStencilOp(GL_KEEP,GL_KEEP,GL_REPLACE);

glBindVertexArray(0);
glStencilFunc(GL_EQUAL, 1, 0xFF);
glStencilOp(GL_KEEP,GL_REPLACE,GL_REPLACE);

// draw the things that are visible through this window. The window is defined by m_screenQuad

glDisable(GL_STENCIL_TEST);


This works great.

The problem I'm having arises when one of the things that are visible through the window, are also another window.

In the above example you can see that I didn't mask out the red channel, this was to reveal that when the bounds of an "interior" container went outside the edge of he exterior container, the whole stencil doesn't work as planned.

Here's some screen shots with the above code (again, note that the red it to show the different bounds being drawn by the m_screenQuad objects).

Here's another image where the object inside the first Container is a subsequent Container. Each calls the same code above, with the second one being called at the // draw the things that are visible through this window. The window is defined by m_screenQuad line.

As you can see, the bounding m_screenQuad being drawn in the same way by the internal Container overlaps with the external Containers m_screenQuad and effectively extends the Stencil Mask further than it should.

How do I separate these two different Stencil tests so that they don't affect each other? Consider that the internal Container may be smaller than the external one, so it would need to truncate/cut it's internal contents as well.

If all you're using are rectangular areas, consider using glScissor instead.

It is a lot faster as it clips primitives rather than test them per pixel reducing the number of primitives, pixel shader executions, and framebuffer pixels processed.

Stencil test is done at the pixel level, Scissor test is done at the primitive level before the pixel shader.

More complex clipping shapes can be processed using a geometry shader which is still faster than using the stencil buffer.

Otherwise, use all bits of the stencil buffer incrementing the value every time and test for equality.

int stencil_index = 0;

foreach(ui layer){

int ui_layer_stencil_index;

if(ui shape is complex) {
ui_layer_stencil_index = ++stencil_index;

if((stencil_index & ((1 << STENCIL_BITS)-1)) == 0){
// we warped around, clear stencil to 0
ui_layer_stencil_index = stencil_index = 1;
glDisable( GL_SCISSOR_TEST );
glClear(GL_STENCIL_BUFFER_BIT);
}
glStencilMask(-1u);    // everything is written to the buffer
glStencilFunc(GL_ALWAYS, ui_layer_stencil_index, -1u);
glStencilOp(GL_KEEP,GL_KEEP,GL_REPLACE);
}

glScissor( MAX RECTANGULAR UI LAYER AREA HERE );
glEnable( GL_SCISSOR_TEST );

...

// draw ui layer area here
...

if(ui shape is complex) {
glStencilFunc(GL_EQUAL, ui_layer_stencil_index, -1u);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);
...
}

...

foreach(elements on this layer)
// draw objects on this UI layer


Note that this does not handle recursive clipping shapes, you can't properly handle all cases with this simple stencil test, you'll have to clear the stencil and process the clipping hierarchy every time.

If that is required it becomes more efficient to calculate the clipped geometry right on the CPU and submit batches of pre-clipped & ordered triangles to the GPU or generate a clipping shape on the CPU from the hierarchy and use the geometry shader to clip to that shape but there is a limit to the shape's complexity that can be processed.

• Thanks, I have simple rectangles to clip in this instance, I'll look into using glScissor or the geometry shader, I hadnt thought of using anything else. Can you explain what a recursive stencil test is? – NeomerArcana Dec 31 '14 at 23:11
• If a parent element A clip child element B which then clips child element C you need to use the union of all clipping zones. – Stephane Hockenhull Jan 1 '15 at 0:01
• Yep, that was the trick. Really simple intersect solution. Thanks a lot, you just improved my framerate by about 300fps. – NeomerArcana Jan 1 '15 at 4:59

If you need separate stencil tests, you have 8-bits worth of stencil to fool around with. Your code is not accumulating stencil values, it only performs binary set/clear, so that means you can do up to 8 boolean tests per-pixel with the proper masking logic.

Rather than testing for equality versus 1 using the mask 0xff, you can use a more sophisticated mask to isolate the individual bits. The mask 0x2 and 2 for instance will test the 2nd bit, while 0x1 and 1 will test the 1st. Right now you are using the entire 8-bit stencil buffer to do a single boolean test, and that is not going to help you at all.

• Thanks for your response. Do I understand you correctly that I would draw the first container with 1's and then test against the 1's for the second shape, drawing the second shape with 3's etc? Is it ref values I'm changing or the masks im using? – NeomerArcana Dec 31 '14 at 23:15
• You want to use both, change the ref value in order to set the bit and change the mask so that you ignore the contents of all the other bits during the test. A value of 3 is 11 in binary, and that would mean that both bit 1 and 2 were set. The idea of the mask is to tell GL which bits you are interested in. If you want bit 3, instead of 3 you would use 0x4 (100 in binary). – Andon M. Coleman Jan 1 '15 at 0:14
• Could you make a small pseudo code example? I have problems gasping it. – clankill3r Aug 17 '19 at 18:32