# Stencil based mask with alpha

I'm trying to create a fog effect like civ 5:

I tried creating a StencilSet shader that receives a mask and sets the stencil buffer to 1. I would render the following on every unexplored tile using this shader. White is full alpha, black is 0 alpha, gray is in between.

Then I created a StencilUse shader that would display the cloud as one big texture, but will test that stencil value is 1. This way the cloud will only be shown on top of the unexplored tiles (+ a bit of overlap outside the tile, per the mask sprite).

This is what I got:

The problem is that I can't produce the fading effects towards the end of the cloud. It's either fully visible or not visible at all. How would I be able to use the grey areas when determining the alpha value of the cloud?

Any ideas? (I'm using Unity)

• I don't think the Civ V example is using the stencil buffer for this. See this answer some ideas on how to implement fog of war in Unity. (Note that the comment on Unity Pro is out of date. RenderTextures can now be used in the free version of Unity) Jul 6, 2015 at 16:52

Using only stencil will not help you if you want smooth fading of your clouds. Stencil (as the name suggests) is used to reject pixels which are masked out by it.

When you draw your b/w alpha tile you probably setup stencil test so that if alpha!=0 then stencil passes, otherwise - fails. So you get what you asked for.

When you later draw your clouds they will be hard-cutted according to the stencil mask.

I do not know all details of your code, but it could be easier to just draw alpha-transparent clouds only over unexplored tiles, but also multiply them by your b/w color tile. This way your unexplored tiles will have fading borders. Drawback is that there will be fading borders also between any 2 consequent unexplored tiles. To avoid this you can have several types of b/w textures (depending on type of neighbour tiles), but then you'll have to issue separate draw call for each tile (or unique configuration, if you sort them by b/w texture). I am not familiar with Unity, so unfortunately cannot say how to exactly do it there. But if you got the idea, then it should be easy.

More proper solution would be to generate distance field texture.

• The problem with drawing smaller clouds on unexplored tiles (which was our original concept), is that it creates a very tiled look, whatever we tried. The fading is not a problem because we made the cloud image a bit larger than the tile, and the outside part faded. We use the same technique when using mask sprites. We ended up writing a shader that sets the alpha of the display buffer depending on the alpha of the mask sprite, and then another shader that draws the cloud sprite and blending it using 1 minus destination alpha. Works nicely, you can see it in www.suncrash.com Jul 7, 2015 at 8:14

What you could do is to set the alpha by using a formula similar to this:

If(length(color) != 0)

Alpha = 1 - length(normalize(color));

Else

Alpha = 0

• The problem is that the mask is not actually rendered in the color buffer, it's only used to set the stencil, so I don't have it's color or alpha value when the cloud is rendered. Jun 6, 2015 at 14:59