Stencil buffer mask calculation

Question

I dont understand how exactly stencil buffer works in openGL. One aspect that confuses me is why do we use a bit operator there in glStencilFunc. Some text say that is is used to achieve multiple stencil planes but I am not sure how that can be done. Can anybody please explain the whole procedure of stencil buffer calculation during the rendering cycle. Thank you.

Answer 1

Okay, this does sound like something you can do with stenciling. Stenciling is generally used in conjunction with Multipass Rendering.

Multipass Rendering

Multipass Rendering basically refers to a rendering loop where multiple renders a performed per frame. Each pass may use a different configuration of OpenGL (e.g. with different stenciling settings) and render the same or different artifacts. The pseudo code would look something like this:

initialize the OpenGL configuration for the entire sequence

for each frame
{
    initialize the OpenGL configuration for the frame

    for each rendering pass
    {
        initialize the OpenGL configuration for the rendering pass

        render the artifacts
    }
}

In your case, you will need a rendering pass for each layer of rectangles so that you can prepare the stencil buffer before each pass.

Your Problem

Effectively, you want to restrict the pixels that can be drawn in each rendering pass to the set of pixels that were drawn in the previous rendering pass. To do this, you are going to need to do three things:

1. Record the pixels you draw.
2. Only draw pixels that were also drawn in the previous rendering pass (except for the first rendering pass, of course).
3. Before each frame, remove the record of which pixels have been drawn.

Record the pixels you draw

This is where glStencilOp() comes in, glStencilOp() determines what's going to happen to the stencil buffer when a pixel is drawn (and in other situations too, but we won't worry about that). To increment the value in the stencil buffer for each pixel drawn (and hence record that you drew it) include this in the initialization of the entire sequence:

glStencilOp(GL_KEEP, GL_KEEP, GL_INCR);

The third argument is the important one, it determines what happens when a pixel is drawn (normally), to understand the function better, read the reference docs.

Only draw pixels that were also drawn in the previous rendering pass

This is a job for glStencilFunc(). It determines what's going to happen to the frame buffer depending on what the value in the stencil buffer is.

You need to call it before every rendering pass to say 'only draw if the stencil buffer says I drew last time round'. Since the stencil buffer is being incremented each time, the value to test against needs to be incremented too. For example, the first three rendering passes should call glStencilFunc() like this:

glStencilFunc(GL_EQUAL, 0, 1); // if (stencil_buffer_value == 0) draw;
glStencilFunc(GL_EQUAL, 1, 1); // if (stencil_buffer_value == 1) draw;
glStencilFunc(GL_EQUAL, 2, 1); // if (stencil_buffer_value == 2) draw;

Just ignore the third parameter for now, read the reference documentation if want to know what it is.

EDIT: The third parameter is used when you want to do bit-wise tests against the stencil buffer value. Most implementations allow 8 bits in the stencil buffer (otherwise known as 8 planes). The third parameter allows you to mask out bits that you do not want to include in the test (by ANDing it with the second parameter before doing the test). Since I have used 1 as the third parameter, ANDing it has no effect. Also, since we are treating the stencil buffer as holding integers in this example, bit-wise tests don't make sense. By using these planes, though, we could potentially hold 8 stencils in the buffer at one time and choose which combination of stencils we want to use in each rendering pass.

Remove the record of which pixels have been drawn

Each frame you don't want to continue adding to the existing stencil buffer values just like you don't want to leave the previously rendered image in the frame buffer. You clear it in the same way you would clear the frame buffer too:

glClear(GL_STENCIL_BUFFER_BIT);

Or, do both at once:

glClear(GL_FRAME_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);