We are into a game where to build our tile map, we carve on an image with a mask (At first the whole map is an image, and we create holes on it with tiles to create the map and le the background show. It is like substracting instead of adding new tiles in any common tiled based game).

We have done this with just a tile that acts as a mask using stencil buffer. The problem comes when we want to mask with tiles where all their pixels are not opaque, for example a slope. The problem is that it applies the mask as if all the pixels in tile where opaque and we want to apply the mask in two differnet ways:

1) Keep the pixels of the big image where the mask pixels are opaque

2) Mask the pixels of the big image where the mask pixels are transparent (alpha==0)

I have read a lot of info about using stencil and alpha blending but can't make it work. Checking the pipeline order it seems that stencil is applied after the fragment color has been calculated. Is it possible to access the alpha value at stencil stage and write to the stencil as needed to just mask part of the image (tile in my case).

Update to add a visual example: enter image description here

  • \$\begingroup\$ Can you include an image or animation of the final effect you want to achieve? There may be easier routes to this than selective stencil writing. \$\endgroup\$
    – DMGregory
    Commented Nov 22, 2016 at 16:32
  • \$\begingroup\$ @DMGregory I have added an image explaining the process we are following. We are choosing this workflow because it gives artists a lot of freedom when creating platforms (they can paint a full image or several without the need of using tiles). Then just carve on it what they need. \$\endgroup\$
    – Notbad
    Commented Nov 22, 2016 at 19:47

1 Answer 1


Well, I finally fixed this using a discard for the pixels I don't want to be tested by stencil. It is working but discard operations are not the most performant :). Depending on the user case it could be a solution. It is in mine.



You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .