I have a simple 2D java game engine running using LWJGL. I have my own vertex and fragment shaders, and I'm able to render quads with textures.

I want to be able to mask one quad using the texture from another quad. Example: I have two quads with two different textures mapped to them. Quad A has a red circle texture. Quad B has a blue square texture. I want to use Quad A's red circle texture as a mask such that Quad B will only be visible if it is within Quad A's red circle texture.

Here's an image showing what I want to do:

mask example

The problem is that I am always seeing the 'full overlap' no matter the position of the quads. I can never get that partial masking that I'm looking for.

My shader logic is close to this:

#version 150 core
uniform sampler2D texture;
uniform sampler2D maskTexture;

in vec2 pass_TextureCoord;
in vec2 pass_MaskTextureCoord;

out vec4 out_Color;

void main(void) {
     //***other logic here to determine out_Color***

     if(pass_MaskTextureCoord.x != -1 && pass_MaskTextureCoord.y != -1) { //if we have valid mask texture coordinates, we are masking
          vec4 maskColor = texture(maskTexture, pass_MaskTextureCoord); //get the rgba values of the mask texture
          if(maskColor.a == 0) { //if mask texture has no alpha, discard

Do I need to pass location data of the masking quad into the shader? If so, how do I do that?

EDIT: ilmale's method using the stencil buffer worked for me! All I needed to add was a uniform boolean value to tell the shader if I was drawing a mask or not.

Here's the relevant shader code:

if(isMask) { //if this is a mask, discard where texture alpha is 0
    if(texColor.a == 0) {
    } else {
        out_Color = vec4(0, 0, 0, 0); //make the rest transparent, so we don't see the mask texture

this is the classical use of stencil buffer.

glStencilFunc( GL_ALWAYS, 1, 0xFF );

glStencilOp( GL_KEEP, GL_KEEP, GL_KEEP );
glStencilFunc( GL_EQUAL, 1, 0xFF );

QuadA will write the stencil buffer to 1. QuadB will be drawn only where the stencil buffer is 1.

  • \$\begingroup\$ Stencil does not work for my application because I want to mask based on the texture inside the quad, not the quad itself. I read that there was a way to combine stenciling and GL_ALPHA_TEST to achieve my effect, but alpha testing has been deprecated in OpenGL 3.0+, and I am using OpenGL 3.2 \$\endgroup\$ – Mark D. Feb 23 '15 at 18:52
  • \$\begingroup\$ In the pixel shader of quadA you can discard the pixel that don't need to go on the stencil based on the texture. \$\endgroup\$ – ilmale Feb 24 '15 at 0:27
  • \$\begingroup\$ Huh. Never thought of trying to discard fragments from Quad A instead of Quad B. I'll give this a try. \$\endgroup\$ – Mark D. Feb 25 '15 at 13:17
  • \$\begingroup\$ Works perfectly now! I'll accept your answer and update my question with the relevant changes. Thanks. =) \$\endgroup\$ – Mark D. Feb 25 '15 at 16:33

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.