Need to combine a color, mask, and sprite layer in a shader

Question

My task: to display a sprite using different team colors. I have a sprte graphic, part of which has to be displayed as a team color. The color isn't 'flat', i.e. it shades from brighter to darker. I can't "pre-build" the graphics because there are just too many, so I have to generate them at runtime. I've decided to use a shader, and supply it with a texture consisting of the team color, a texture consisting of a mask (black=no color, white=full color, gray=progressively dimmed color), and the sprite grapic, with the areas where the team color shows being transparent.

So here's my shader code:

// Effect attempts to merge a color layer, a mask layer, and a sprite layer
// to produce a complete sprite

sampler UnitSampler : register(s0); // the unit
sampler MaskSampler : register(s1); // the mask
sampler ColorSampler : register(s2);    // the color

float4 main(float4 color : COLOR0, float2 texCoord : TEXCOORD0) : COLOR0
{
    float4 tex1 = tex2D(ColorSampler, texCoord);    // get the color
    float4 tex2 = tex2D(MaskSampler, texCoord); // get the mask
    float4 tex3 = tex2D(UnitSampler,texCoord);  // get the unit
    float4 tex4 = tex1 * tex2.r * tex3; // color * mask * unit

    return tex4;
}

My problem is the calculations involving tex1 through tex4. I don't really understand how the manipulations work, so I'm just thrashing around, producing lots of different incorrect effects. So given tex1 through tex3, what calcs do I do in order to take the color (tex1), mask it (tex2), and apply the result to the unit if it's not zero?

And would I be better off to make the mask just on/off (white/black) and put the color shading in the unit graphic?

Answer 1

Might I suggest that the simplest (and so probably the best) solution is not to use a custom shader at all. Simply have two sprites.

The first sprite can be your "base" layer containing parts of the unit common to all teams. This can be colour with an alpha channel.

And the second layer would contain only the team-coloured elements in grayscale with an alpha channel that then gets colour tinted by the default sprite batch shader.

spriteBatch.Draw(baseTexture, position, Color.White);
spriteBatch.Draw(teamOverlayTexture, position, teamColor);

---

However, if you have your heart set on using a multi-texture shader:

Your first issue seems to be that you're trying to have your mask texture provide two things at once - both transparency and value. My reading of your question is that you want the value of 0 on your mask texture to be fully transparent. But a value like 0.1 to be "fully opaque and almost black".

So the first thing to do would be to make your mask texture's black-and-white colour values provide the level of grayscale "shadowing". Then give your mask an alpha channel to provide the transparency values for what areas of the unit have colour applied.

Once you have done this, this pixel shader code (that I haven't tested) should give you the right result (I've renamed your variables for better clarity - you should do the same):

float4 outColor;
outColor.rgb = teamColor.rgb * mask.rgb + baseTexture.rgb;
outColor.a = mask.a + baseTexture.a;

Actually code like the following would be better, I wanted to show you how to access sub-elements. If you ensure the team colour is always opaque, then this works:

outColor = teamColor * maskColor + baseTexture;

Note #1: This assumes both your mask and baseTexture textures have premultiplied alpha, and you are using a premultiplied alpha output blend. This is the default in XNA 4.

Note #2: teamColor doesn't have to come from a texture. You should probably provide it via a shader parameter.

(Of course, the textures used and the result is identical to the sprite-batch code above. So I'd go with that unless you're going to do something fancier in the shader.)