How to implement input-texture limited alphablending of 2 textures with HLSL?

Question

I try to implement a HLSL shader the does the normal Alphablend with premultiplied colors (just as XNA4 does) but depending on some existing colors. One can think of adding a glow to a 2D terrain where the glow should only be applied to the not fully transparent parts of the terrain.

My current code is:

sampler TextureSampler1 : register(s1); // Addition (Terrain-only particles)
sampler TextureSampler2 : register(s2); // Existing (Terrain)

float4 main(float4 color : COLOR0, float2 texCoord : TEXCOORD0) : COLOR0
{
    // Look up color from partly transparent existing texture (premultipied colors)
    float4 existing = tex2D(TextureSampler2, texCoord);

    // For transparent pixels -> Color stays the same
    if ((existing.r + existing.g + existing.b + existing.a) == 0) return existing;

    // Look up color from partly transparent addition-texture (e.g. glow) (using premultipied colors) that should be added to existing color
    float4 addition = tex2D(TextureSampler1, texCoord);

    // Rescale light that is added by existing alpha
    addition.rgb *= existing.a;

    // Use this color to blend
    existing.rgb *= (1 - addition.a) + addition.rgb; 

    return existing;
}

However, there still seems to be a problem with the alpha channel. Any ideas?

Answer 1

Your line:

existing.rgb *= (1 - addition.a) + addition.rgb; 

does

existing.rgb = existing.rgb * ((1 - addition.a) + addition.rgb);

what you probably wanted is

existing.rgb = existing.rgb * (1 - addition.a) + addition.rgb;

Answer 2

Interestingly, the following code works. But why?

float4 main(float4 color : COLOR0, float2 texCoord : TEXCOORD0) : COLOR0
{
    // Look up color from partly transparent existing texture (premultipied colors)
    float4 existing = tex2D(TextureSampler2, texCoord);

    // For transparent pixels -> Color stays the same
    if ((existing.r + existing.g + existing.b + existing.a) == 0) return existing;

    // Look up color from partly transparent addition-texture (e.g. glow) (premultipied colors) that should be added to existing color
    float4 addition = tex2D(TextureSampler1, texCoord);

    float4 output;

    // Keep light blocking properties
    output.a = existing.a; 
    // Rescale light that is added by existing alpha
    addition.rgb = addition.rgb * existing.a;
    // Use this color to blend
    output.rgb = existing.rgb * (1 - addition.a) + addition.rgb; 

    return output;
}