# Best way to mask 2D sprites in XNA?

I currently am trying to mask some sprites. Rather than explaining it in words, I've made up some example pictures:

The area to mask (in white)

Now, the red sprite that needs to be cropped.

The final result.

Now, I'm aware that in XNA you can do two things to accomplish this:

1. Use the Stencil Buffer.

I have tried to do a pixel shader, which essentially did this:

float4 main(float2 texCoord : TEXCOORD0) : COLOR0
{
float4 tex = tex2D(BaseTexture, texCoord);

{
return float4(tex.r, tex.g, tex.b, tex.a);
}
else
{
return float4(0, 0, 0, 0);
}
}


This seems to crop the images (albeit, not correct once the image starts to move), but my problem is that the images are constantly moving (they aren't static), so this cropping needs to be dynamic.

Is there a way I could alter the shader code to take into account it's position?

Alternatively, I've read about using the Stencil Buffer, but most of the samples seem to hinge on using a rendertarget, which I really don't want to do. (I'm already using 3 or 4 for the rest of the game, and adding another one on top of it seems overkill)

The only tutorial I've found that doesn't use Rendertargets is one from Shawn Hargreaves' blog over here. The issue with that one, though is that it's for XNA 3.1, and doesn't seem to translate well to XNA 4.0.

It seems to me that the pixel shader is the way to go, but I'm unsure of how to get the positioning correct. I believe I would have to change my onscreen coordinates (something like 500, 500) to be between 0 and 1 for the shader coordinates. My only problem is trying to work out how to correctly use the transformed coordinates.

Thanks in advance for any help!

• Stencil seems to be the way to go, though, I need to know how to properly use them too :P I'll be waiting for a good answer on the matter. – Gustavo Maciel Oct 2 '12 at 18:00
• Heh, most of the good Stencil tutorials are for XNA 3.1, and most of the 4.0 ones use RenderTargets (which seems wasteful, to me). I've updated my question with a link to an old 3.1 tutorial that seemed like it could work, but not in 4.0. Maybe someone knows how to translate it to 4.0 correctly? – electroflame Oct 2 '12 at 18:05
• If you would use pixel shader to accomplish it, I think you would have to unproject your pixel coordinate to screen/world(depends on what you want to do), and this is kinda wasteful(stencil would not have this problem) – Gustavo Maciel Oct 2 '12 at 18:08
• This question is an excellent question. – ClassicThunder Oct 3 '12 at 4:45

You will also need to add some parameters to it that inform the pixel shader where you want your stencil to be located.

sampler BaseTexture : register(s0);
};

//All of these variables are pixel values
//Feel free to replace with float2 variables
float BaseTextureLocationX;  //This is where your texture is to be drawn
float BaseTextureLocationY;  //texCoord is different, it is the current pixel
float BaseTextureWidth;
float BaseTextureHeight;

float4 main(float2 texCoord : TEXCOORD0) : COLOR0
{
//We need to calculate where in terms of percentage to sample from the MaskTexture
float maskPixelX = texCoord.x * BaseTextureWidth + BaseTextureLocationX;
float maskPixelY = texCoord.y * BaseTextureHeight + BaseTextureLocationY;

float4 tex = tex2D(BaseTexture, texCoord);

//It is a good idea to avoid conditional statements in a pixel shader if you can use math instead.
//Alternate calculation to invert the mask, you could make this a parameter too if you wanted
//return tex * (1.0 - bitMask.a);
}


In your C# code you pass in variables to the pixel shader prior to the SpriteBatch.Draw call like this:

maskEffect.Parameters["MaskWidth"].SetValue(800);

• Thank you for the answer! This seems like it will work, but currently I'm having an issue with it. Currently, it's being cropped on the left of the image (with a straight edge, as well). Am I supposed to be using screen coordinates for the positions? Or am I already supposed to have converted them to 0 - 1? – electroflame Oct 3 '12 at 1:24
• I see the problem. I should have compiled the pixel shader before sharing it. :P In the maskCoord calculation one of the X's should have been a Y. I've edited my initial answer with the fix and included the UV clamp settings you will need for your MaskTexture sampler. – Jim Oct 3 '12 at 4:40
• Now it works perfectly, thank you! The only thing that I should mention is that if you have your sprites centered (using Width/2 and Height/2 for the origin) you will need to subtract half of your Width or Height for your X and Y locations (that you pass into the shader). After that it works perfectly, and is extremely fast! Thanks a ton! – electroflame Oct 3 '12 at 16:54

The first step is to tell the graphics card we need the stencil buffer. To do this when you create GraphicsDeviceManager we set the PreferredDepthStencilFormat to DepthFormat.Depth24Stencil8 so there is actually a stencil to write to.

graphics = new GraphicsDeviceManager(this) {
PreferredDepthStencilFormat = DepthFormat.Depth24Stencil8
};


The AlphaTestEffect is used to set the coordinate system and filter the pixels with alpha that pass the alpha test. We are not going to set any filters and set the coordinate system to the view port.

var m = Matrix.CreateOrthographicOffCenter(0,
graphics.GraphicsDevice.PresentationParameters.BackBufferWidth,
graphics.GraphicsDevice.PresentationParameters.BackBufferHeight,
0, 0, 1
);
var a = new AlphaTestEffect(graphics.GraphicsDevice) {
Projection = m
};


Next we need to set up two DepthStencilStates. These states dictate when the SpriteBatch renders to the stencil and when the SpriteBatch renders to the BackBuffer. We are primarily interested in two variables StencilFunction and StencilPass.

• StencilFunction dictates when the SpriteBatch will draw individual pixels and when they will be ignored.
• StencilPass dictates when drawn pixels pixels effect the Stencil.

For the first DepthStencilState we set StencilFunction to CompareFunction. This causes the StencilTest to succeed and when the StencilTest the SpriteBatch renders that pixel. StencilPass is set to StencilOperation. Replace meaning that when the StencilTest succeed that pixel will written to the StencilBuffer with the value of the ReferenceStencil.

var s1 = new DepthStencilState {
StencilEnable = true,
StencilFunction = CompareFunction.Always,
StencilPass = StencilOperation.Replace,
ReferenceStencil = 1,
DepthBufferEnable = false,
};


In summary the StencilTest always passes, the image is drawn to screen normally, and for pixels drawn to the screen a value of 1 is stored in the StencilBuffer.

The second DepthStencilState is slightly more complicated. This time we want to only draw to the screen when the value in the StencilBuffer is. To achieve this we set the StencilFunction to CompareFunction.LessEqual and the ReferenceStencil to 1. This means that when the value in the stencil buffer is 1 the StencilTest will succeed. Setting StencilPass to StencilOperation. Keep causes the StencilBuffer not to update. This allows us to draw multiple times using the same mask.

var s2 = new DepthStencilState {
StencilEnable = true,
StencilFunction = CompareFunction.LessEqual,
StencilPass = StencilOperation.Keep,
ReferenceStencil = 1,
DepthBufferEnable = false,
};


In summary the StencilTest only passes when the StencilBuffer is less than 1 (the alpha pixels from the mask) and does not effect the StencilBuffer.

Now that we have our DepthStencilStates set up. We can actually draw using a mask. Simply draw the mask using the first DepthStencilState. This will effect both the BackBuffer and the StencilBuffer. Now that the stencil buffer has a value of 0 where you mask had transparency and 1 where it contained color we can use StencilBuffer to mask later images.

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s1, null, a);
spriteBatch.End();


The second SpriteBatch uses the second DepthStencilStates. No matter what you draw, only the pixels where the StencilBuffer is set to 1 will pass the stencil test and be drawn to the screen.

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s2, null, a);
spriteBatch.Draw(color, Vector2.Zero, Color.White); //The background
spriteBatch.End();


Below is the entirety of the code in the Draw method, don't forget to set PreferredDepthStencilFormat = DepthFormat.Depth24Stencil8 in the game constructor.

GraphicsDevice.Clear(ClearOptions.Target
| ClearOptions.Stencil, Color.Transparent, 0, 0);

var m = Matrix.CreateOrthographicOffCenter(0,
graphics.GraphicsDevice.PresentationParameters.BackBufferWidth,
graphics.GraphicsDevice.PresentationParameters.BackBufferHeight,
0, 0, 1
);

var a = new AlphaTestEffect(graphics.GraphicsDevice) {
Projection = m
};

var s1 = new DepthStencilState {
StencilEnable = true,
StencilFunction = CompareFunction.Always,
StencilPass = StencilOperation.Replace,
ReferenceStencil = 1,
DepthBufferEnable = false,
};

var s2 = new DepthStencilState {
StencilEnable = true,
StencilFunction = CompareFunction.LessEqual,
StencilPass = StencilOperation.Keep,
ReferenceStencil = 1,
DepthBufferEnable = false,
};

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s1, null, a);
spriteBatch.End();

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s2, null, a);
spriteBatch.Draw(color, Vector2.Zero, Color.White); //The background
spriteBatch.End();

• +1 This is one of the most complete StencilBuffer examples I've seen for XNA 4.0. Thanks for this! The only reason I chose the pixel shader answer is because it was easier to set up (and actually faster to boot), but this is a perfectly valid answer as well and may be easier if you already have a lot of shaders in place. Thanks! Now we've got a complete answer for both routes! – electroflame Oct 3 '12 at 16:52
• This is the best real solution for me , and more than the first – Mehdi Bugnard Oct 17 '13 at 14:06
• How could i use that if i had a 3D model ? I think this could solve my question gamedev.stackexchange.com/questions/93733/… , but i dont know how to apply it on 3D models. Do you know if i can do that ? – dimitris93 Feb 10 '15 at 2:44

In the answer above, some weird things happened when I did exactly as is explained.

The only thing I needed was the both DepthStencilStates.

var s1 = new DepthStencilState {
StencilEnable = true,
StencilFunction = CompareFunction.Always,
StencilPass = StencilOperation.Replace,
ReferenceStencil = 1,
DepthBufferEnable = false,
};

var s2 = new DepthStencilState {
StencilEnable = true,
StencilFunction = CompareFunction.LessEqual,
StencilPass = StencilOperation.Keep,
ReferenceStencil = 1,
DepthBufferEnable = false,
};


And the draws without the AlphaTestEffect.

spriteBatch.Begin(SpriteSortMode.Immediate, null, null, s1, null, null);