# Trying to use stencils in 2D while retaining layer depth

This is a screen of what's going on just so you can get a frame of reference.

The problem I'm running into is that my game is slowing down due to the amount of texture swapping I'm doing during my draw call. Since walls, characters, floors, and all objects are on their respective sprite sheet containing those types, per tile draw, the loaded texture is swapping no less than 3 to 5+ times as it cycles and draws the sprites in order to layer properly.

Now, I've tried throwing all common objects together into their respective lists, and then using layerDepth drawing them that way which makes things a lot better, but the new problem I'm running into has to do with the way my doors/windows are drawn on walls. Namely, I was using stencils to clear out a block on the walls that are drawn in the shape of the door/window so that when the wall would draw, it would have a door/window sized hole in it.

This is the way my draw was set up for walls when I was going tile by tile rather than grouped up common objects.

1. first it would check to see if a door/window was on this wall. If not, it'd skip all the steps and just draw normally. Otherwise
2. end the current spriteBatch
3. Clear the buffers with a transparent color to preserve what was already drawn
4. start a new spritebatch with stencil settings
5. draw the door area
6. end the spriteBatch
7. start a new spritebatch that takes into account the previously set stencil
8. draw the wall which will now be drawn with a hole in it
9. end that spritebatch
10. start a new spritebatch with the normal settings to continue drawing tiles

In the tile by tile draw, clearing the depth/stencil buffers didn't matter since I wasn't using any layerDepth to organize what draws on top of what.

Now that I'm drawing from lists of common objects rather than tile by tile, it has sped up my draw call considerably but I can't seem to figure out a way to keep the stencil system to mask out the area a door or window will be drawn into a wall. The root of the problem is that when I end a spriteBatch to change the DepthStencilState, it flattens the current RenderTarget and there is no longer any depth sorting for anything drawn further down the line. This means walls always get drawn on top of everything regardless of depth or positioning in the game world and even on top of each other as the stencil has to happen once for every wall that has a door or window.

Does anyone know of a way to get around this? To boil it down, I need a way to draw having things sorted by layer depth while also being able to stencil/mask out portions of specific sprites.

• I don't know enough about XNA to post a real answer, but this seems suspicious: "when I end a spriteBatch...it flattens the current RenderTarget and there is no longer any depth sorting". This shouldn't be happening if the sprites are actually being drawn into the depth buffer (I'm assuming that's what "layerDepth" does); changing depth-stencil state needn't clear the depth buffer. So it might be worth looking into whether there's a way to change that behavior. – Nathan Reed Feb 4 '12 at 21:00
• When you say "start a new spritebatch" you don't mean you're making a new SpriteBatch instance, do you? Creating a new batch every time you draw something would slow down your game quite a bit. – Mike Cluck Mar 9 '12 at 0:33

I don't know enough XNA to give you a proper solution, but I can give you a general answer.

The problem you're having is the painter's algorithm. Specifically, you need to order all the things you want to draw before you can draw them. The most common way to avoid this problem is with a depth-buffer. Interestingly, this is also called a z-buffer, because it stores the z-values in camera space.

However, because you don't have Z (because you are writing a 2D game), you cannot use this technique. Or can you?

Consider this: an orthographic projection doesn't change the perspective of points in 3D points. This is useful for things like user interface elements, because you always want them to be the same ways.

What I suggest is that you write a custom shader that takes 3D matrices and a depth value per vertex. This depth value should be used for depth buffering, but not for projection.

[vertex shader]