I dont understand how exactly stencil buffer works in openGL. One aspect that confuses me is why do we use a bit operator there in glStencilFunc. Some text say that is is used to achieve multiple stencil planes but I am not sure how that can be done. Can anybody please explain the whole procedure of stencil buffer calculation during the rendering cycle. Thank you.

• Are you looking for a basic explanation of how the stencil buffer works or something more specific? – Gyan aka Gary Buyn Jun 17 '11 at 5:36
• Let me explain with the following example. I want to draw a rectangle which might contain more rectangles and they too might contain more rectangles and this goes on. So you can have series of rectangles inside a rectangle. Suppose if there is a property which allow me to clip the contents of a rectangle, then child rectangles should not be drawn outside the bounds of the parent rectangle. Now how can I achieve this using a stencil test. Basically how can I have series of clips inside another clip? – Vijayendra Jun 17 '11 at 18:20
• wouldn't scissor be more fit for what you are trying to achieve? – diego.martinez Nov 23 '16 at 12:39

Okay, this does sound like something you can do with stenciling. Stenciling is generally used in conjunction with Multipass Rendering.

## Multipass Rendering

Multipass Rendering basically refers to a rendering loop where multiple renders a performed per frame. Each pass may use a different configuration of OpenGL (e.g. with different stenciling settings) and render the same or different artifacts. The pseudo code would look something like this:

initialize the OpenGL configuration for the entire sequence

for each frame
{
initialize the OpenGL configuration for the frame

for each rendering pass
{
initialize the OpenGL configuration for the rendering pass

render the artifacts
}
}


In your case, you will need a rendering pass for each layer of rectangles so that you can prepare the stencil buffer before each pass.

Effectively, you want to restrict the pixels that can be drawn in each rendering pass to the set of pixels that were drawn in the previous rendering pass. To do this, you are going to need to do three things:

1. Record the pixels you draw.
2. Only draw pixels that were also drawn in the previous rendering pass (except for the first rendering pass, of course).
3. Before each frame, remove the record of which pixels have been drawn.

### Record the pixels you draw

This is where glStencilOp() comes in, glStencilOp() determines what's going to happen to the stencil buffer when a pixel is drawn (and in other situations too, but we won't worry about that). To increment the value in the stencil buffer for each pixel drawn (and hence record that you drew it) include this in the initialization of the entire sequence:

glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);


The third argument is the important one, it determines what happens when a pixel is drawn (normally), to understand the function better, read the reference docs.

### Only draw pixels that were also drawn in the previous rendering pass

This is a job for glStencilFunc(). It determines what's going to happen to the frame buffer depending on what the value in the stencil buffer is.

You need to call it before every rendering pass to say 'only draw if the stencil buffer says I drew last time round'. Since the stencil buffer is being incremented each time, the value to test against needs to be incremented too. For example, the first three rendering passes should call glStencilFunc() like this:

glStencilFunc(GL_EQUAL, 0, 1); // if (stencil_buffer_value == 0) draw;
glStencilFunc(GL_EQUAL, 1, 1); // if (stencil_buffer_value == 1) draw;
glStencilFunc(GL_EQUAL, 2, 1); // if (stencil_buffer_value == 2) draw;


Just ignore the third parameter for now, read the reference documentation if want to know what it is.

EDIT: The third parameter is used when you want to do bit-wise tests against the stencil buffer value. Most implementations allow 8 bits in the stencil buffer (otherwise known as 8 planes). The third parameter allows you to mask out bits that you do not want to include in the test (by ANDing it with the second parameter before doing the test). Since I have used 1 as the third parameter, ANDing it has no effect. Also, since we are treating the stencil buffer as holding integers in this example, bit-wise tests don't make sense. By using these planes, though, we could potentially hold 8 stencils in the buffer at one time and choose which combination of stencils we want to use in each rendering pass.

### Remove the record of which pixels have been drawn

Each frame you don't want to continue adding to the existing stencil buffer values just like you don't want to leave the previously rendered image in the frame buffer. You clear it in the same way you would clear the frame buffer too:

glClear(GL_STENCIL_BUFFER_BIT);


Or, do both at once:

glClear(GL_FRAME_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);

• Huge help! Thanks Gary. I really appreciate this detailed explanation. There is still little confusion in my mind though. If you see my original question, I was not sure about why do we need a mask value (third parameter) in glStencilFunc. I am not sure how does that help? I have read openGL documents, but I am not able to realize why do I ever need it? Thanks again for the help. – Vijayendra Jun 18 '11 at 9:32
• @Vijayendra See my edit. – Gyan aka Gary Buyn Jun 18 '11 at 22:15
• Thanks again for the details. So, does that mean for every pixel in framebuffer, there is an integer in the stencil buffer PLUS 8 bits (representing 8 levels of filter)? I believe answer to this question will clear the situation as not too many texts clearly explain this. Appreciate you time. Cheeers! – Vijayendra Jun 19 '11 at 8:48
• @Vijayendra No problem. I haven't worked with stencil buffers for a few months so it helped me to revisit them :) For every pixel in the frame buffer, there are (usually) 8 bits in the stencil buffer. You can use them however you like. In our example we used them to represent integers (8 bits can be used to represent the numbers between 0 and 255). Another use of them is to create 8 planes, 1 bit per plane. It depends on what you're trying to achieve. But no, there is not an integer AND 8 bits, there is only 8 bits. – Gyan aka Gary Buyn Jun 19 '11 at 10:50
• Oh! Now that makes everything clear. Now I understand. That make sense. The flexibility of using 8 bits was a source of confusion. I am quite disappointed with the way various texts explain this issue. Yours explanation was the better then many professional authors. Thanks again Gary! – Vijayendra Jun 19 '11 at 15:42