Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

Environment

This is the environment I am working in:

  • OpenGL ES 2.0
  • iPhone Simulator & iPhone 4
  • iMac 27" using NVIDIA GeForce GTX 680MX 2048 MB

Hope that helps.

The Problem

Been searching high and low from multiple sources and multiple sites including Stackoverflow but haven't got a working stencil effect going.

What I have is this:

stencil scene setup

The black 'S' thing is not a polygon but a texture drawn onto a rectangle quad that is the same width and height as the background image.

I am trying to do a stencil effect where the background and the little yellow guy should only be visible if it's within that black 'S' texture.

In my fragment shader, I have this:

varying lowp vec4 destinationColor;

varying lowp vec2 TexCoordOut;
uniform sampler2D Texture;
uniform highp float TexScale;

void main()
{
    highp vec4 color = texture2D(Texture, TexCoordOut);

    if(color.a == 0.0)
    {
        discard;
    }

    gl_FragColor = color;
}

For my setup Depth Stencil Buffer, I have set it up like this:

-(void)setupDepthStencilBuffer
{
    GLint width;
    GLint height;

    glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_WIDTH, &width);
    glGetRenderbufferParameteriv(GL_RENDERBUFFER, GL_RENDERBUFFER_HEIGHT, &height);

    glGenBuffers(1, &depthStencilBuffer);
    glBindRenderbuffer(GL_RENDERBUFFER, depthStencilBuffer);
    glRenderbufferStorage(GL_RENDERBUFFER, GL_DEPTH24_STENCIL8_OES, width, height);
    //glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_COMPONENT24_OES, width, height);

    NSLog(@"depthStencilBuffer = %d", depthStencilBuffer);
}

According to Apple's documentation (which I think is outdated):

http://developer.apple.com/library/ios/#documentation/3DDrawing/Conceptual/OpenGLES_ProgrammingGuide/WorkingwithEAGLContexts/WorkingwithEAGLContexts.html#//apple_ref/doc/uid/TP40008793-CH103-SW1

Note some of the things like GL_RGBA enum doesn't exist when I tried to type it in Xcode (I think Apple must have removed it and made it obsolete).

I have also tried Apple's way of defining the so called "depth/stencil" buffer in the above link but it gave the same error below.

The code I have above is how you would create a stencil buffer.

In my setupFrameBuffer() method, I attach it like so:

-(void)setupFrameBuffer
{
    GLuint frameBuffer;

    glGenBuffers(1, &frameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, colorRenderBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, depthStencilBuffer);

    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

    if(status != GL_FRAMEBUFFER_COMPLETE)
    {
        NSLog(@"failed to make complete framebuffer object %x", status);
    }
}

The error I got when I attach it like shown above is:

failed to make complete framebuffer object 8cd6

For my render method, I have this:

-(void)render:(CADisplayLink *)displayLink
{
    glClearColor(0.5f, 0.5f, 0.5f, 1.0f);
    glClear(GL_COLOR_BUFFER_BIT | GL_STENCIL_BUFFER_BIT);



    glClearStencil(0);
    glEnable(GL_STENCIL_TEST);

    // ----------------------------------------
    // Don't write to color or depth buffer
    // ----------------------------------------
    glColorMask(GL_FALSE, GL_FALSE, GL_FALSE, GL_FALSE);
    glDepthMask(GL_FALSE);

    glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

    // ----------------------------------------
    // First set the alpha test so that
    // fragments greather than threshold
    // will pass thus will set nonzero
    // bits masked by 1 in stencil
    // ----------------------------------------
    glStencilFunc(GL_ALWAYS, 1, 1);

    // ----------------------------------------------------------------
    // Drawing our stencil
    // ----------------------------------------------------------------

    glBindBuffer(GL_ARRAY_BUFFER, bgVBO);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, bgIBO);

    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, stencilTexture);
    glUniform1i(textureUniform, 0);

    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));
    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_FAN, sizeof(bgIndices)/sizeof(bgIndices[0]), GL_UNSIGNED_BYTE, 0);

    /*
    // -----------------------------------------
    // Second pass of the fragments less
    // or equal than the threshold will pass
    // thus will set zero bits masked by 1
    // in stencil
    // -----------------------------------------
    glStencilFunc(GL_ALWAYS, 0, 1);

    glBindBuffer(GL_ARRAY_BUFFER, bgVBO);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, bgIBO);

    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, stencilTexture);
    glUniform1i(textureUniform, 0);

    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));
    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_STRIP, sizeof(bgIndices)/sizeof(bgIndices[0]), GL_UNSIGNED_BYTE, 0);
    */


    // ---------------------------------------------------
    // RE-ENABLING THE COLOR AND DEPTH MASK AGAIN
    // TO DRAW REST OF THE SCENE AFTER STENCIL
    // ---------------------------------------------------
    glColorMask(GL_TRUE, GL_TRUE, GL_TRUE, GL_TRUE);
    glDepthMask(GL_TRUE);




    Mat4 frustrumMatrix = [CameraMatrix createOrthographicMatrixUsingLeft:-(self.bounds.size.width / 2.0)
                                                                       Right:(self.bounds.size.width / 2.0)
                                                                      Bottom:-(self.bounds.size.height / 2.0)
                                                                         Top:(self.bounds.size.height / 2.0)
                                                                        Near:-1.0f
                                                                         Far:1.0f];

    glUniformMatrix4fv(projectionUniform, 1, 0, frustrumMatrix.matrix);

    glViewport(0, 0, self.bounds.size.width * self.contentScaleFactor, self.bounds.size.height * self.contentScaleFactor);

    // ----------------------------------------------------------------
    // Drawing our background first
    // ----------------------------------------------------------------

    glBindBuffer(GL_ARRAY_BUFFER, bgVBO);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, bgIBO);

    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, bgTexture);
    glUniform1i(textureUniform, 0);

    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));
    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_FAN, sizeof(bgIndices)/sizeof(bgIndices[0]), GL_UNSIGNED_BYTE, 0);



    glBindBuffer(GL_ARRAY_BUFFER, vertexBuffer);
    glBindBuffer(GL_ELEMENT_ARRAY_BUFFER, indexBuffer);


    glActiveTexture(GL_TEXTURE0);
    glBindTexture(GL_TEXTURE_2D, floorTexture);
    glUniform1i(textureUniform, 0);


    glVertexAttribPointer(positionSlot, 3, GL_FLOAT, GL_FALSE, sizeof(Vertex), 0);
    glVertexAttribPointer(colorSlot, 4, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 3));

    glVertexAttribPointer(texCoordSlot, 2, GL_FLOAT, GL_FALSE, sizeof(Vertex), (const void *)(sizeof(GLfloat) * 7));

    glDrawElements(GL_TRIANGLE_FAN, sizeof(Indices)/sizeof(Indices[0]), GL_UNSIGNED_BYTE, 0);

    glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);
    glEnable(GL_BLEND);

    [context presentRenderbuffer:GL_RENDERBUFFER];
}

The result of this is obviously a pink screen that means my setup is wrong:

enter image description here

Anyone able to shed some light please ?

share|improve this question
    
No idea how to solve the exact problem, but why not just use the "stencil texture" while drawing the little yellow guy, and discard if the texel value doesn't match? –  Jari Komppa Jan 8 '13 at 14:02
    
The stencil is not for masking the little guy, it's kinda for use in detecting terrain that the little guy stands on (not the background image you see above). Think of Worm game, you got the background, then you got the terrain in the middle, that is masked by a stencil to both erase the terrain and to detect collision for the players on screen (the little yellow guy). –  Zhang Jan 8 '13 at 14:13
add comment

1 Answer

up vote 5 down vote accepted

The Solution

Hell yeah!!!

I'm one happy chap now! :D

OK, I finally manage to get Stencil working with texture :)

(also learned a number of things along the way, e.g. we can check color.alpha and use discard as a way to remove transparent pixel and the glBlend(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA) trick becomes obsolete)

So the first thing I noticed I did wrong was incorrectly generating the names for my stencil buffer.

In the above source code in my question, I had typed:

glGenBuffers(1, &depthStencilBuffer);

This should really be:

glGenRenderbuffers(1, &depthStencilBuffer);

d'oh!

Second of all, I commented out the important second glStencilFunc() that needed to be called:

glStencilFunc(GL_ALWAYS, 1, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_REPLACE);

// draw black 'S' textured quad here

. . .

// -----------------------------------
// I WAS MISSING THIS IMPORTANT LINE
// -----------------------------------
glStencilFunc(GL_ALWAYS, 0, 1);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

/* draw the rest of the scene here (the background and yellow guy) */

The final result:

enter image description here

Hope that helps everyone else who is trying this cool stencil with a texture feature :D

I also modified a number of other source code but those two were the main changes that got it working.

Some helpful changes helped me debug the problem too:

  • I have found that I don't need to attach depth buffer to my framebuffer to get it to work
  • I also don't need to clear GL_DEPTH_BUFFER_BIT
  • I changed the setupFrameBuffer() method to this with the extra if-else conditions to clearly point to me the possible errors.

The new setupFrameBuffer() method:

-(void)setupFrameBuffer
{
    GLuint frameBuffer;

    glGenBuffers(1, &frameBuffer);
    glBindFramebuffer(GL_FRAMEBUFFER, frameBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_COLOR_ATTACHMENT0, GL_RENDERBUFFER, colorRenderBuffer);
    //glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_DEPTH_ATTACHMENT, GL_RENDERBUFFER, depthStencilBuffer);
    glFramebufferRenderbuffer(GL_FRAMEBUFFER, GL_STENCIL_ATTACHMENT, GL_RENDERBUFFER, depthStencilBuffer);

    GLenum status = glCheckFramebufferStatus(GL_FRAMEBUFFER);

    if(status == GL_FRAMEBUFFER_COMPLETE)
    {
        NSLog(@"framebuffer complete");
        //NSLog(@"failed to make complete framebuffer object %x", status);
    }
    else if(status == GL_FRAMEBUFFER_INCOMPLETE_ATTACHMENT)
    {
        NSLog(@"incomplete framebuffer attachments");
    }
    else if(status == GL_FRAMEBUFFER_INCOMPLETE_MISSING_ATTACHMENT)
    {
        NSLog(@"incomplete missing framebuffer attachments");
    }
    else if(status == GL_FRAMEBUFFER_INCOMPLETE_DIMENSIONS)
    {
        NSLog(@"incomplete framebuffer attachments dimensions");
    }
    else if(status == GL_FRAMEBUFFER_UNSUPPORTED)
    {
        NSLog(@"combination of internal formats used by attachments in thef ramebuffer results in a nonrednerable target");
    }
}
share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.