I have a simple 2D java game engine running using LWJGL. I have my own vertex and fragment shaders, and I'm able to render quads with textures.

I want to be able to mask one quad using the texture from another quad. Example: I have two quads with two different textures mapped to them. Quad A has a red circle texture. Quad B has a blue square texture. I want to use Quad A's red circle texture as a mask such that Quad B will only be visible if it is within Quad A's red circle texture.

Here's an image showing what I want to do:

mask example

The problem is that I am always seeing the 'full overlap' no matter the position of the quads. I can never get that partial masking that I'm looking for.

My shader logic is close to this:

#version 150 core
uniform sampler2D texture;
uniform sampler2D maskTexture;

in vec2 pass_TextureCoord;
in vec2 pass_MaskTextureCoord;

out vec4 out_Color;

void main(void) {
     //***other logic here to determine out_Color***

     if(pass_MaskTextureCoord.x != -1 && pass_MaskTextureCoord.y != -1) { //if we have valid mask texture coordinates, we are masking
          vec4 maskColor = texture(maskTexture, pass_MaskTextureCoord); //get the rgba values of the mask texture
          if(maskColor.a == 0) { //if mask texture has no alpha, discard

Do I need to pass location data of the masking quad into the shader? If so, how do I do that?

EDIT: ilmale's method using the stencil buffer worked for me! All I needed to add was a uniform boolean value to tell the shader if I was drawing a mask or not.

Here's the relevant shader code:

if(isMask) { //if this is a mask, discard where texture alpha is 0
    if(texColor.a == 0) {
    } else {
        out_Color = vec4(0, 0, 0, 0); //make the rest transparent, so we don't see the mask texture

1 Answer 1


this is the classical use of stencil buffer.

glStencilFunc( GL_ALWAYS, 1, 0xFF );

glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
glStencilFunc( GL_EQUAL, 1, 0xFF );

QuadA will write the stencil buffer to 1. QuadB will be drawn only where the stencil buffer is 1.

  • \$\begingroup\$ Stencil does not work for my application because I want to mask based on the texture inside the quad, not the quad itself. I read that there was a way to combine stenciling and GL_ALPHA_TEST to achieve my effect, but alpha testing has been deprecated in OpenGL 3.0+, and I am using OpenGL 3.2 \$\endgroup\$
    – Mark D.
    Commented Feb 23, 2015 at 18:52
  • \$\begingroup\$ In the pixel shader of quadA you can discard the pixel that don't need to go on the stencil based on the texture. \$\endgroup\$
    – ilmale
    Commented Feb 24, 2015 at 0:27
  • \$\begingroup\$ Huh. Never thought of trying to discard fragments from Quad A instead of Quad B. I'll give this a try. \$\endgroup\$
    – Mark D.
    Commented Feb 25, 2015 at 13:17
  • \$\begingroup\$ Works perfectly now! I'll accept your answer and update my question with the relevant changes. Thanks. =) \$\endgroup\$
    – Mark D.
    Commented Feb 25, 2015 at 16:33

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .