Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I'm currently building a top down 2D tile based game using OpenGL. There are three tile layers, the background layer, player/sprite layer & foreground layer. The foreground layer is pretty much used to place translucent tiles that give the impression of a shadow when blended over the two underlying layers. The problem is that many textures for tiles in the background layer already have a shadow applied in their source images. I would like to exclude these background tiles from having the shadow overlaid whilst still applying it to sprites that may be above these tiles. Any ideas as to how I could go about accomplishing this? In the accompanying image notice how I only want the forground shadow applied to the character in the sprite layer to get the result.

Desired result

share|improve this question
    
Do you mean transparent? Translucent is much more complex. –  Pubby Nov 5 '11 at 20:36
    
A screenshot would be helpful. –  Tetrad Nov 5 '11 at 20:39
    
So you only want the shadow overlay layer to apply to non-transparent pixels in the player layer? Or are you trying to make it such that you don't have "double shadows" on the background layer? –  Tetrad Nov 5 '11 at 20:40
    
I've added a screenshot to clarify but yes essentially I want to avoid the double shadows you speak of. –  dee Nov 5 '11 at 21:34
    
There are several ways of doing what you describe. However, your picture doesn't seem to be showing off what you're talking about. In the "desired result", the foreground shadowing does affect the background (just not as much in the "current result"), which is not what you said you wanted. You said that you wanted the shadowing to only affect the sprites. Could you clarify a bit more the difference between what you've shown us you want and what you say you want? –  Nicol Bolas Nov 5 '11 at 21:48
show 2 more comments

1 Answer

up vote 3 down vote accepted

The following answer will assume that you are trying to do the following:

There are certain tiles in your background which already have shadows baked into their images. You don't want your shadowing layer to affect them at all, but you do want your shadowing layer to affect everything else.

There are several ways to accomplish this.

Multitexture

No "foreground layer" at all. Instead, have each tile tell whether it is shadowed or not. Render the non-shadowed tiles as normal. The shadowed tiles are rendered with two textures and two texture coordinates: one that picks from the background texture atlas (or however you're doing your rendering) and one that picks from the foreground atlas. You simply do the "layering" as part of your shader/texture environment. It's a simple multiplication, or perhaps an alpha blend, depending on how your shadowing works.

The same goes for sprites.

This method only works on a per-tile granularity.

Stenciling

You can employ a stencil buffer (if you do, make sure you also have a depth buffer. 24/8 depth/stencil in size). This brings layering back, but you still need to mark which background tiles need to be shadowed and which ones don't.

At the beginning of the frame, clear the color, depth, and stencil buffers. The stencil buffer should be cleared to 0.

Tiles that do not need shadows should be rendered with a stencil value of 0. Tiles and sprites that need them should be rendered with a stencil of 1. Make sure that your sprites use alpha tests/discards to cut out the area around them.

The code for this would be (warning: untested):

glEnable(GL_STENCIL_TEST);
//Change the second parameter to 0x01 for objects that should be shadowed.
glStencilFunc(GL_ALWAYS, 0x00, 0x01);
glStencilOp(GL_NEVER, GL_REPLACE, GL_REPLACE);

When you draw your shadowing layer, what you do is turn on the stencil test and have the test function be equals. All of your foreground should be rendered with a stencil value of 1. The code you would use before rendering your shadowing layer would be (warning: untested):

glEnable(GL_STENCIL_TEST);
glStencilFunc(GL_EQUAL, 0x01, 0x01);
glStencilOp(GL_KEEP, GL_KEEP, GL_KEEP);

Again, this has a per-tile granularity.

share|improve this answer
    
Thanks for the detailed answer! I used your Stenciling approach as it resulted in the least amount of change to my renderer. –  dee Nov 6 '11 at 22:52
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.