Trying to use stencils in 2D while retaining layer depth

Question

This is a screen of what's going on just so you can get a frame of reference.

The problem I'm running into is that my game is slowing down due to the amount of texture swapping I'm doing during my draw call. Since walls, characters, floors, and all objects are on their respective sprite sheet containing those types, per tile draw, the loaded texture is swapping no less than 3 to 5+ times as it cycles and draws the sprites in order to layer properly.

Now, I've tried throwing all common objects together into their respective lists, and then using layerDepth drawing them that way which makes things a lot better, but the new problem I'm running into has to do with the way my doors/windows are drawn on walls. Namely, I was using stencils to clear out a block on the walls that are drawn in the shape of the door/window so that when the wall would draw, it would have a door/window sized hole in it.

This is the way my draw was set up for walls when I was going tile by tile rather than grouped up common objects.

1. first it would check to see if a door/window was on this wall. If not, it'd skip all the steps and just draw normally. Otherwise
2. end the current spriteBatch
3. Clear the buffers with a transparent color to preserve what was already drawn
4. start a new spritebatch with stencil settings
5. draw the door area
6. end the spriteBatch
7. start a new spritebatch that takes into account the previously set stencil
8. draw the wall which will now be drawn with a hole in it
9. end that spritebatch
10. start a new spritebatch with the normal settings to continue drawing tiles

In the tile by tile draw, clearing the depth/stencil buffers didn't matter since I wasn't using any layerDepth to organize what draws on top of what.

Now that I'm drawing from lists of common objects rather than tile by tile, it has sped up my draw call considerably but I can't seem to figure out a way to keep the stencil system to mask out the area a door or window will be drawn into a wall. The root of the problem is that when I end a spriteBatch to change the DepthStencilState, it flattens the current RenderTarget and there is no longer any depth sorting for anything drawn further down the line. This means walls always get drawn on top of everything regardless of depth or positioning in the game world and even on top of each other as the stencil has to happen once for every wall that has a door or window.

Does anyone know of a way to get around this? To boil it down, I need a way to draw having things sorted by layer depth while also being able to stencil/mask out portions of specific sprites.

Answer 1

I don't know enough XNA to give you a proper solution, but I can give you a general answer.

The problem you're having is the painter's algorithm. Specifically, you need to order all the things you want to draw before you can draw them. The most common way to avoid this problem is with a depth-buffer. Interestingly, this is also called a z-buffer, because it stores the z-values in camera space.

However, because you don't have Z (because you are writing a 2D game), you cannot use this technique. Or can you?

Consider this: an orthographic projection doesn't change the perspective of points in 3D points. This is useful for things like user interface elements, because you always want them to be the same ways.

What I suggest is that you write a custom shader that takes 3D matrices and a depth value per vertex. This depth value should be used for depth buffering, but not for projection.

In pseudo-shadercode:

[vertex shader]
pos_output = TransformOrthographic * TransformView * vec3(pos_vertex.x, pos_vertex.y, 0.0);
depth_output = depth_vertex;

[fragment shader]
depth_output = (depth_input - depth_min) / (depth_max - depth_min);

Once you have this set up, you can stop worrying about objects being rendered in the wrong order and can instead set their depth value.

Now you should be able to render everything in one draw call, without worrying about stenciling out sections of the screen.