Stencil mask with AlphaTestEffect

Question

I am trying to pull off the following effect in XNA 4.0:

The purple area has 50% opacity. I have gotten pretty close with the following code:

public static DepthStencilState AlwaysStencilState = new DepthStencilState()
        {

            StencilEnable = true,
            StencilFunction = CompareFunction.Always,
            StencilPass = StencilOperation.Replace,
            ReferenceStencil = 1,
            DepthBufferEnable = false,
        };

public static DepthStencilState EqualStencilState = new DepthStencilState()
        {
            StencilEnable = true,
            StencilFunction = CompareFunction.Equal,
            StencilPass = StencilOperation.Keep,
            ReferenceStencil = 1,
            DepthBufferEnable = false,
        };

...

if (_alphaEffect == null)
{
       _alphaEffect = new AlphaTestEffect(_spriteBatch.GraphicsDevice);
       _alphaEffect.AlphaFunction = CompareFunction.LessEqual;
       _alphaEffect.ReferenceAlpha = 129;
       Matrix projection = Matrix.CreateOrthographicOffCenter(0, _spriteBatch.GraphicsDevice.PresentationParameters.BackBufferWidth, _spriteBatch.GraphicsDevice.PresentationParameters.BackBufferHeight, 0, 0, 1); 
       _alphaEffect.Projection = world.SystemManager.GetSystem<RenderSystem>().Camera.View * projection;
}

_mask = new RenderTarget2D(_spriteBatch.GraphicsDevice, _spriteBatch.GraphicsDevice.PresentationParameters.BackBufferWidth, _spriteBatch.GraphicsDevice.PresentationParameters.BackBufferHeight, false, SurfaceFormat.Color, DepthFormat.Depth24Stencil8);
_spriteBatch.GraphicsDevice.SetRenderTarget(_mask);
_spriteBatch.GraphicsDevice.Clear(ClearOptions.Target | ClearOptions.Stencil, Color.Transparent, 0, 0); 

_spriteBatch.Begin(SpriteSortMode.Immediate, null, null, AlwaysStencilState, null, _alphaEffect);
_spriteBatch.Draw(sprite.Texture, position, sprite.SourceRectangle,Color.White, 0f, sprite.Origin, 1f, SpriteEffects.None, 0);
_spriteBatch.End();

_spriteBatch.Begin(SpriteSortMode.Immediate, null, null, EqualStencilState, null, null);
_spriteBatch.Draw(_maskTex, new Vector2(x * _maskTex.Width, y * _maskTex.Height), null, Color.White, 0f, Vector2.Zero, 1f, SpriteEffects.None, 0);
_spriteBatch.End();

_spriteBatch.GraphicsDevice.SetRenderTarget(null);
_spriteBatch.GraphicsDevice.Clear(Color.Black);

_spriteBatch.Begin();
_spriteBatch.Draw((Texture2D)_mask, Vector2.Zero, null, Color.White, 0f, Vector2.Zero, 1f, SpriteEffects.None, layer/max_layer);
_spriteBatch.End();

My problem is, I can't get the AlphaTestEffect to behave. I can either mask over the semi-transparent purple junk and fill it in with the green design, or I can draw over the completely opaque grassy texture. How can I specify the exact opacity that needs to be replace with the green design?

Answer 1

I'm not sure if you can do this with the AlphaTestEffect(in fact I does not know how ATE works). I recently want to draw some kind of menu, that fills the the the stencil buffer correctly and is a kind of mask.

My PixelShader looks like this:

float4 PSFunc (VertexShaderOutput input) : COLOR0
{
    float4 texel = tex2D(TextureSampler, input.TexCoord);
    clip(texel.w - AlphaClipValue);
    return texel;
}

AlphaClipValue defines what pixel are visible in texture. e.g. 0.8f defines that alle Alpha values in the texture that are above 204 gets drawn. Other pixel get cliped.

protected readonly DepthStencilState DrawMaskDepthStencilState = 
    new DepthStencilState(){
        StencilEnable = true,
        StencilFunction = CompareFunction.Always,
        StencilPass = StencilOperation.Invert,
        ReferenceStencil = 0,
        DepthBufferEnable = true,
    };
protected readonly DepthStencilState DrawSceneDepthStencilState = 
    new DepthStencilState(){
    StencilEnable = true,
        StencilFunction = CompareFunction.Equal,
        StencilPass = StencilOperation.Keep,
        ReferenceStencil = 0,
        DepthBufferEnable = true,
    };

Now the Draw()

public override void Draw(GameTime gameTime){
    //set state that is able to draw
    GraphicsDevice.DepthStencilState = DrawMaskDepthStencilState;

    //set texture and AlphaClipValue (1 means only pixel with Alpha==255 gets drawn)
    DrawMenuEffect.Texture = _maskTexture;
    DrawMenuEffect.AlphaClipValue = 1f;
    DrawMenuEffect.CurrentTechnique.Passes[0].Apply();
    GraphicsDevice.DrawUserPrimitives(PrimitiveType.TriangleStrip,    FullScreenQuadStripVertices,0,2);

    GraphicsDevice.DepthStencilState = DrawSceneDepthStencilState;

    // Now draw rest of the scene 
}

I using my own projected quads to draw the texture, because I do not like the SpriteBatch(it resets some state that is does not reset). But I think it should work with SpriteBatch, too. Otherwise you can look here to see how to create them.