Stencil based mask with alpha

Question

I'm trying to create a fog effect like civ 5:

I tried creating a StencilSet shader that receives a mask and sets the stencil buffer to 1. I would render the following on every unexplored tile using this shader. White is full alpha, black is 0 alpha, gray is in between.

Then I created a StencilUse shader that would display the cloud as one big texture, but will test that stencil value is 1. This way the cloud will only be shown on top of the unexplored tiles (+ a bit of overlap outside the tile, per the mask sprite).

This is what I got:

The problem is that I can't produce the fading effects towards the end of the cloud. It's either fully visible or not visible at all. How would I be able to use the grey areas when determining the alpha value of the cloud?

Any ideas? (I'm using Unity)

Answer 1

Using only stencil will not help you if you want smooth fading of your clouds. Stencil (as the name suggests) is used to reject pixels which are masked out by it.

When you draw your b/w alpha tile you probably setup stencil test so that if alpha!=0 then stencil passes, otherwise - fails. So you get what you asked for.

When you later draw your clouds they will be hard-cutted according to the stencil mask.

I do not know all details of your code, but it could be easier to just draw alpha-transparent clouds only over unexplored tiles, but also multiply them by your b/w color tile. This way your unexplored tiles will have fading borders. Drawback is that there will be fading borders also between any 2 consequent unexplored tiles. To avoid this you can have several types of b/w textures (depending on type of neighbour tiles), but then you'll have to issue separate draw call for each tile (or unique configuration, if you sort them by b/w texture). I am not familiar with Unity, so unfortunately cannot say how to exactly do it there. But if you got the idea, then it should be easy.

More proper solution would be to generate distance field texture.

Answer 2

What you could do is to set the alpha by using a formula similar to this:

If(length(color) != 0)

Alpha = 1 - length(normalize(color));

Else

Alpha = 0