I'm writing a transparency shader where I need any intersecting textures to, wherever they overlap, take the texture pixel with the smallest alpha value and render only that, discarding the other pixel(s).

Turn this:

enter image description here

Into this:

enter image description here

In pseudocode, something along the lines of…

if ( newPixel.alpha < currentPixelInFrameBuffer.alpha )
    currentPixelInFrameBuffer = newPixel;

I can't seem to find if there is a native alpha blending command that will do this. There exists DstAlpha, which is an alpha blending property that compares against the current fragment's frame buffer source alpha value, but I obviously cannot use that in the following manner…

AlphaTest Less DstAlpha

I've found there's a BlendOp Min, which is supposed to "Use the smaller of source and destination." but I can't seem to get that to work and I can't find documentation on what it means by 'smaller'.

  • 1
    \$\begingroup\$ BlendOp min performs a component-wise min operation. For example, if my frame buffer currently contains a pixel value of (0.1, 0.2, 0.4, 0.8) and I perform a min blend with the fragment (0.3, 0.3, 0.3, 0.3) (assuming Blend One One), I'll get (0.1, 0.2, 0.3, 0.3) - the r & g components of the background are kept, and the b & a components are replaced. So, I don't think a direct min in the blend pipeline will be possible. Can you describe more about the effect you want to achieve? There's almost certainly another way to tackle it. \$\endgroup\$ – DMGregory Mar 15 '14 at 3:23
  • \$\begingroup\$ Thanks for the reply @DMGregory. What I'm trying to do is create a 2D fog of war effect in my level screen. I position the above textures to create transparent holes through the fog. But where they intersect I want them to not blend, but merge so that they form a path through the fog. More detail and pics available: stackoverflow.com/questions/22383582/… \$\endgroup\$ – Essential Mar 15 '14 at 13:39

To get the RGB values of your scene influenced by the minimum of several alpha values from your fog, the most straightforward way to use a render texture. This requires Unity Pro.

[Edit: Unity 5 now allows the use of render textures in the free version as well. Yay! :D ]

(Some alternatives are provided below if this isn't an option)

Set up a second camera, with Culling Mask set so that it sees only your fog hole quads. Set its Background to the colour you want your fog to be, and set its Depth so that it renders before the rest of your scene.

Then place a quad with this texture in front of your scene, so that it's visible to your main camera, and it should render on top as expected.

Set your fog hole shader to use:

BlendOp Min
Blend One One

and output all-white for the colour channel. (Either by using a .tga for your fog hole texture that's all white on the colour planes, or via the shader). This ensures the holes will never modify the fog colour, only its alpha.

Alternatively, you can make the RenderTexture in code with only a single channel, and apply any colour adjustment you want when you blend the finished fog into your scene. The advantage of rendering it as a full-colour map is mainly Editor convenience, and the ability to render arbitrary content into it like roiling clouds for the holes to later punch-out.

The reason compositing directly to your frame buffer without an intermediate texture won't work is that, as soon as that first quad is drawn, the RGB values of your scene in the dark region have already been modified. A quad drawn over it later with a lighter patch can't undo that change (at least not without dividing by DstAlpha, which is not one of the available operations).

If you're not able to use RenderTextures, your next best bets are:

  1. changing your visual style so the fog isn't one even shade. That way you can just stamp down dark fog quads everywhere outside the main view, rather than punching holes inside it.
  2. using the Mesh class to construct a one-layer piece of geometry that can be rendered for your fog without the same texel being blended over multiple times. One straightforward way to do this is a grid with vertex colours indicating visible/obscured cells, and a fragment shader that uses a noise texture to bias this value to keep the edge rough & feathery.

Edit: here's an example of suggestion #2, quickly mocked-up with an 8x8 vertex grid blended over a happy face reference texture.

Fog of war makes me happy

  • \$\begingroup\$ Sorry for the late reply but my HDD failed! I'll try your other suggested approach of stamping fog around the areas, rather than inside them, as I am on Unity basic. \$\endgroup\$ – Essential Apr 8 '14 at 17:44
  • \$\begingroup\$ No worries. I mocked up the second suggestion in the free version of Unity above - you can see it gives a consistent level of opacity in the dark regions while still allowing irregular feathering at the edges. \$\endgroup\$ – DMGregory Apr 10 '14 at 0:51
  • 1
    \$\begingroup\$ That mesh approach actually looks really a lot better than I anticipated! No chance you could give me a headstart with the mockup you made, is there? :) \$\endgroup\$ – Essential Apr 10 '14 at 19:13
  • 1
    \$\begingroup\$ Also, I should've checked your profile before! I just realized we've met. \$\endgroup\$ – Essential Apr 10 '14 at 19:13
  • 1
    \$\begingroup\$ I was leading the team that sat behind you at TOJam 2011 — we made the zombies attacking a house game. I also live around Lansdowne so I've seen you around there too. Big coincidence, eh? Hehe. I added you on Facebook! I'm sure we can figure it out from there. Thanks :) \$\endgroup\$ – Essential Apr 10 '14 at 19:56

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.