XNA/MonoGame/SharpDX: Pixel shader with sprite sheet

Question

I've searched around for two days now on the internet but cannot find a solution. I've also read up on Pixel Shaders on MSDN with no luck.

I'm trying to apply a simple pixel shader to a sprite batch for alpha masking. It worked fine when I used two separate textures; one for the sprite and a matching one for the shader.

Problem is I don't want to do this. I'd like to use my sprite sheet and source rectangle and use a separate mask from a sprite sheet of masks.

However it doesn't work. The sprite draw perfectly but the mask does not and appears to be using a sprite at a different locating in the mask sheet. It has something to do with getting the coordinates of the mask.

My idea was if the sprite to draw was at 200*200 in the sheet, subtract the correct tex coordinates from it to get the pixel, then add this to the mask.

pos = texCoord - sprite start = offset + mask start = location of mask from current pixel

EG; pos = 200*200 - 200*200 = 0 + maskPos = 440*330 pos = 201*201 - 200*200 = 1 + maskPos = 441*330

UPDATE: It is definitely converting the coordinates of the sprite to the mask that is the issue. I've tried numerous methods to try and convert this and normalize the coordinates to singles with no luck.

1. The result after masking
2. The mask being applied
3. The actual mask that should be applied

I have no idea why the result is being botched and section cut out like shown in image one. 22 is the width and height of both masks and sprites. 512 is the width and height of both sprite sheets.

To calculate the pos in the mask that generate the botch:

float maskPixelX = (inCoord.x * 512) + X pos to draw the tile;
float maskPixelY = (inCoord.y * 512) + X pos to draw the tile;
float2 maskCoord = float2((maskPixelX - maskpos.x) / 22, (maskPixelY - maskpos.y) / 22);

Pixel shader:

uniform extern texture ScreenTexture;  
sampler screen = sampler_state 
{

    Texture = <ScreenTexture>;
};


uniform extern texture MaskTexture;  
sampler mask = sampler_state
{
    Texture = <MaskTexture>;
};

float2 startpos;
float2 maskpos;


float4 PixelShaderFunction(float2 inCoord: TEXCOORD0) : COLOR
{

    // Get the matching pixel offset in mask.
    float2 pos = inCoord - startpos;
    pos = pos + maskpos;

    float4 color = tex2D(screen, inCoord);
    float4 color2 = tex2D(mask, pos);
    if (color2.a > 0) color.rgba = 0;
    return color;
}

technique
{
    pass P0
    {
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}

And here is the draw call:

  mask.Parameters("MaskTexture").SetValue(maskSheet) ' Sprite sheet with mask.
  mask.Parameters("startpos").SetValue(New Vector2(200, 200))
  mask.Parameters("maskpos").SetValue(New Vector2(440, 330)) ' Postion of mask in sheet.
  Dim source As New Rectangle(200, 300, 22, 22)
  spriteBatch.Draw(sprites, Vector2.Zero, source, Color.White, 0.0F, New Vector2(0, 0), 1.0F, SpriteEffects.None, 1.0F)

Answer 1

It took quite a while but I've found the solution and am posting it here for anyone else who has the same issue.

The draw code remains the same.

Pixel Shader:

// Texture sprite sheet.
uniform extern texture ScreenTexture;
sampler screen : register(s0) = sampler_state
{
    Texture = <ScreenTexture>;
    MinFilter = POINT; // Stops pixels bleeding together.
    MagFilter = POINT;
};

// Alpha map (mask) texture to blend with.
uniform extern texture MaskTexture;
sampler mask : register(s1) = sampler_state
{
    Texture = <MaskTexture>;
    MinFilter = POINT;
    MagFilter = POINT;
};

float2 spritePos; // Screen coordinates of the tile.
float2 maskPos; // Screen coordinates of the mask.

float4 PixelShaderFunction(float2 inCoord: TEXCOORD0) : COLOR
{

    // Convert the mask position to 0-1 texels.
    float2 maskTexel = (maskPos - spritePos) / 512;
    float2 pos = inCoord + maskTexel;

    // Blend!
    float4 color = tex2D(screen, inCoord);
    float4 maskColor = tex2D(mask, pos);
    if (maskColor.a > 0) color.rgba = 0;
    return color;

}

technique
{
    pass P0
    {
        PixelShader = compile ps_2_0 PixelShaderFunction();
    }
}