We are into a game where to build our tile map, we carve on an image with a mask (At first the whole map is an image, and we create holes on it with tiles to create the map and le the background show. It is like substracting instead of adding new tiles in any common tiled based game).
We have done this with just a tile that acts as a mask using stencil buffer. The problem comes when we want to mask with tiles where all their pixels are not opaque, for example a slope. The problem is that it applies the mask as if all the pixels in tile where opaque and we want to apply the mask in two differnet ways:
1) Keep the pixels of the big image where the mask pixels are opaque
2) Mask the pixels of the big image where the mask pixels are transparent (alpha==0)
I have read a lot of info about using stencil and alpha blending but can't make it work. Checking the pipeline order it seems that stencil is applied after the fragment color has been calculated. Is it possible to access the alpha value at stencil stage and write to the stencil as needed to just mask part of the image (tile in my case).
Update to add a visual example:
