# Implementing fog of war in opengl es 2.0 game

Hi game development community, this is my first question here! ;)

I'm developing a tactics/strategy real time android game and I've been wondering for some time what's the best way to implement an efficient and somewhat nice looking fog of war to incorporate in it.

The style of fog of war i'm looking for is to darken the areas where there is no visibility at a certain moment. [not the solid black, unexplored area, kind of fog of war] Note that this has nothing to do with which units are visible to the player. This is merely a visual effect.

My experience with OpenGL or Android is not vast by any means, but I think it is sufficient for what I'm asking here.

So far I have thought in some solutions:

1. Draw white circles to a dark background, corresponding to the units visibility, then render to a texture, and then drawing a quad with that texture with blend mode set to multiply. Will this approach be efficient? Will it take too much memory? (I don't know how to render to texture and then use the texture. Is it too messy?)

2. Have a grid object with a vertex shader which has an array of uniforms having the coordinates of all units, and another array which has their visibility range. The number of units will very probably never be bigger then 100. The vertex shader needs to test for each considered vertex, if there is some unit which can see it. In order to do this it, will have to loop the array with the coordinates and do some calculations based on distance. The efficiency of this is inversely proportional to the looks of it. A more dense grid will result in a more beautiful fog of war... but will require a greater amount of vertexes to be checked. Is it possible to find a nice compromise or is this a bad solution from the start?

Which solution is the best? Are there better alternatives? Which ones?

Edit

I have (kinda) tried the second option with only one unit, and the shadows were getting too pixelated.. If I increased the number of vertexes, the fps started to drop. I tried doing the tests in the fragment shader and the results were better in terms of smoothness (that was expected) but also in terms of performance (comparing to the denser grid), which got me a little surprised. But I don't know if testing each pixel against 50 units might be viable to do in the fragment shader. Probably not!

I'm gonna bring a third option that I don't know if is valid, but maybe you people can enlighten me:

• 3.Use the stencil buffer in some way to draw the invisible areas in white (this would be done by having the stencil buffer initialized in white, and then draw black circle textured quads on it corresponding to units visibility) and then use that to make a kinda "clipping mask" effect on a full black somewhat transparent quad that would darken the invisible areas.

I don't even know if this is possible, and I'm not 100% sure that every android with opengl 2.0 supports stencil buffer. I also don't know very well what is the stencil buffer or when it is used... But apparently this might be a perfect solution. Does anyone have an idea?

• In the second approach! why are you checking all the vertices instead of the units? You can check for all the units for their presence within the circle. – Shashwat Sep 17 '12 at 8:49
• Along with making out of range units invisible, are you shadowing the area too (like in Age of Empires)? – Shashwat Sep 17 '12 at 8:52
• @Slashwat I think you got it wrong. I already got the logic behind which units are visible to the player. What I'm asking here is merely a visual effect [the shadowing you are talking about]. I reformulated the question. – jmacedo Sep 17 '12 at 11:20

I managed to do it with stencil buffer.

Here's how to do it: In GLSurfaceView constructor, put:

setEGLConfigChooser(5,6,5,0,16,8);


before setting the render. This makes sure you have stencil buffer active.

Then in the draw method:

    GLES20.glClear(GLES20.GL_COLOR_BUFFER_BIT | GLES20.GL_STENCIL_BUFFER_BIT);

//draw other unrelated stuff here

GLES20.glEnable(GLES20.GL_STENCIL_TEST);
GLES20.glStencilFunc(GLES20.GL_ALWAYS, 0x1, 0xffffffff);
GLES20.glStencilOp(GLES20.GL_REPLACE, GLES20.GL_REPLACE, GLES20.GL_REPLACE);

for(Unit u: ownUnitsToDraw)
Drawings.drawUnitVisionMask(u); //This is a quad with a texture of a white circle

GLES20.glStencilFunc(GLES20.GL_NOTEQUAL, 1, 1);
GLES20.glStencilOp(GLES20.GL_KEEP, GLES20.GL_KEEP, GLES20.GL_KEEP);

Drawings.drawFog(); //This is a black quad with some transparency

GLES20.glDisable(GLES20.GL_STENCIL_TEST);

//draw other unrelated stuff here


So what happens here is that the black fog is masked to the zone outside the vision of the units which is exactly the intended. If we set glStencilFunc to GLES20.GL_EQUAL it would be the space around the units that would have fog, but that is not wanted.

Also, in order for this to work, the mask quad fragmentation shader needs to be something like this:

    precision mediump float;
uniform sampler2D u_Texture;
varying vec2 vTexCoordinate;
uniform vec4 vColor;
void main() {
gl_FragColor = vColor * texture2D(u_Texture, vTexCoordinate);
if(gl_FragColor.a < 0.5)
};


This will allow the pixels in the quad that are transparent (alpha < 0.5) not to make part of the mask (otherwise the mask would not be circular but squared).

• what about obstacles blocking vision? for example trees or big rocks and those kind of things? – Ali1S232 Sep 30 '12 at 22:36
• @Gajoo You would need to draw some kind of shadow shape in black immediately after drawing units vision mask – jmacedo Sep 30 '12 at 23:49

You can have a vertex grid for the ground.

For every movement of your player, you can check for all the vertices if they are within the range and blend the shadow color. I don't think that would slow down the game even for a dense grid.

Or else just change the Color of only those vertices that have changed. When you move your units, notice the coordinates (based on player position and range) which have entered the range. Use those coordinates as indices to refer to the vertex itself and change its Color.

To get the coordinates which have entered the range, you don't need to check all the vertices.

*----*----*----*----*----*----*
|    |    |    |    |    |    |
|    |    |    |    |    |    |
*----*----*----A----*----*----*
|    |    |    |    |    |    |
|    |    |    |    |    |    |
*----*----A----A----A----*----*
|    |    |    |    |    |    |
|    |    |    |    |    |    |
*----A----A----O----A----A----*
|    |    |    |    |    |    |
|    |    |    |    |    |    |
*----*----A----A----A----*----*
|    |    |    |    |    |    |
|    |    |    |    |    |    |
*----*----*----A----*----*----*
|    |    |    |    |    |    |
|    |    |    |    |    |    |
*----*----*----*----*----*----*

O = player position
A = vertices within range


So if you are at (0, 0) and range is 3, then you can check for vertices between (-3,-3) to (+3, +3).

Note: When you move your unit, some vertices enter the range and some leave it too. So you have to consider both the cases.

Hope this is clear

• Your second suggestion seams a little too far fetched for me but i liked this: "For every movement of your player,you can check for all the vertices if they are within the range and blend the shadow color. I don't think that would slow down the game even for a dense grid." But there are always moving units in my game, so I would be doing this every render?. This may very well be a valid solution, but isn't it equivalent to sol 2 I presented?. However I'm afraid it could drop FPS for the level of detail I want...I'm trying to make a simple,but beautiful game. – jmacedo Sep 17 '12 at 21:18
• I don't know why, I've got the feeling there's simpler solution around the corner I'm not seeing. Do you know anything about the stencil buffer? I don't know if opengl es 2.0 supports it, but in case it does, wouldn't it be possible to draw visibility white circles in it and then somehow use that has a clipping mask for the shadowing ? – jmacedo Sep 17 '12 at 21:23
• I don't know OpenGL. I've worked on XNA. It has a Draw function which gets called automatically. So anyways, it will render all the vertices in each of its call (say after 16ms). In Draw, there is no shorter way. The whole screen (all vertices) gets rendered frequently. Similarly it has an Update function. There you write your update logic. My answer was about this function. What I suggested in the answer was that you don't have to traverse every vertices to set the color. Just check the vertices as suggested in the answer. – Shashwat Sep 18 '12 at 5:04
• Are you creating a game like Age of Empires or some FPS? – Shashwat Sep 18 '12 at 5:07
• It's a RTS, so, like Age of Empires. The thing is in OpenGL changing the color of many individual vertexes one at a time is expensive I think. I know nothing about XNA, but CPU-GPU communication is generally costly. So it might even be more efficient to send the whole chunk of vertices just once even if they keep the same color OR have all the vertexes being tested inside the fragmentation shader. Anyways, I think a pixel oriented solution would fit my purposes better instead of a vertex oriented one. I'll see if I can do something with the stencil buffer! – jmacedo Sep 18 '12 at 13:26