# Dealing with alpha for 2D per pixel lighting

Right now I've got a simple light shader. Every bitmap I draw goes through it to make up the scene. I only draw quads.

   texture tex;
sampler2D s = sampler_state {
texture = <tex>;
};

bool use_tex;

float x = 0.1;
float y = 0.3;

float4 ps_main(VS_OUTPUT Input) : COLOR0
{
float4 p = Input.vPosition / Input.vPosition.w;
float4 color = 0;
float a = color.a;
if(!use_tex)
{
color = Input.Color;
}
else
{
color = Input.Color * tex2D(s, Input.TexCoord.xy);
}
p += 1.0;
p /= 2;
p.y = 1.0 - p.y;

float3 lightAttenuation = float3(0.05,4.0, 0.0);
// float4 color = Input.Color * tex2D(s, Input.TexCoord.xy);
float2 aux = float2(x,y) - p.xy;
float dist = length(aux);
float attenuation=1.0/(lightAttenuation.x+lightAttenuation.y*dist

+lightAttenuation.z*dist*dist);

float3 lightAttenuation2 = float3(0.05,3.0, 0.0);
float2 aux2 = float2(0.5,0.4) - p.xy;
float dist2 = length(aux2);
float attenuation2=1.0/(lightAttenuation2.x

+lightAttenuation2.y*dist2+lightAttenuation2.z*dist2*dist2);

color = (float4(attenuation,attenuation,attenuation,1.0) * color * float4(1.0,0.8,0.8,1.0)

* 0.8)  + (float4(attenuation2,attenuation2,attenuation2,1.0) * color * float4(1.0,0.8,1.0,1.0)

* 0.2);

return color;
}


The shader calculates based on the current pixel's screen position.

The problem is when the pixel's alpha is != 1.0

It this case it would need to be multiplied by what is already in the backbuffer at that pixel location in order to get the correct final pixel.

What can I do to fix the alpha issue and deal with the alpha case?

Thanks

PS2.0 hlsl.

Edit

Here are 2 images, the bright one is when the whole scene is rendered to texture then the shader is applied. The second is when I apply the shader on each fragment as the scene is rendered. This is an area where there is a dark shadow on the ground. The shadow is a semi transparent bitmap. Thus it makes the lighting effect weaker and it makes it look weird.

What I want is a way for it to look like the first one, but still letting me do per fragment shading. What I was saying was that I thought of maybe looking at what is already in the backbuffer, then multiplying it by the current pixel, then taking that result, applying the shader to it, then obviously you set the alpha to 1.0.

Im not sure if or how that could work. Im wondering either how to do that, or a better way to get the same result.

Here is my shader source:

texture tex;
sampler2D s = sampler_state {
texture = <tex>;
};

bool use_tex;

float x = 0.1;
float y = 0.3;

float4 ps_main(VS_OUTPUT Input) : COLOR0
{
float4 p = Input.vPosition / Input.vPosition.w;
float4 color = 0;

if(!use_tex)
{
color = Input.Color;
}
else
{
color = Input.Color * tex2D(s, Input.TexCoord.xy);
}
float a = color.a;
p += 1.0;
p /= 2;
p.y = 1.0 - p.y;

float3 lightAttenuation = float3(0.05,4.0, 0.0);
// float4 color = Input.Color * tex2D(s, Input.TexCoord.xy);
float2 aux = float2(x,y) - p.xy;
float dist = length(aux);
float attenuation=1.0/(lightAttenuation.x+lightAttenuation.y*dist+lightAttenuation.z*dist*dist);

float3 lightAttenuation2 = float3(0.0,2.1, 1.0);
float2 aux2 = float2(-1.5 + x,0.4 + y) - p.xy;
float dist2 = length(aux2);
float attenuation2=1.0/(lightAttenuation2.x+lightAttenuation2.y*dist2+lightAttenuation2.z*dist2*dist2);

float4 c = (float4(attenuation,attenuation,attenuation,1.0) * color * float4(1.0,0.9,0.9,1.0) * 0.2) + (float4(attenuation2,attenuation2,attenuation2,1.0) * color * float4(1.0,0.8,1.0,1.0) * 0.2);

color = c;

color.a = a;

return color;
}

• I'm not sure what you want to achieve - if you want the alpha blending, then just set the alpha value to color.a, the blending will be done after the pixel shader code. If you don't want to blend, then set the alpha to 1.0. Do you have any screens? Aug 23 '12 at 11:04
• What is you actual problem? Is there no blending at all or is it blending in another way than you would like? Just to make sure, did you set the blend state? Aug 23 '12 at 17:25
• @ Mikael Högström see my edit. Aug 23 '12 at 20:30
• The problem is shaders dont let you use the backbuffer to see what is already there and I want to avoid drawing to a texture. Aug 23 '12 at 21:28

You're right, pixel shaders don't let you read from the destination pixel in the render target. That's what the alpha blending stage is for: combining the pixel shader output color with the current color in the destination render target.

The blend stage is configured by the application, not the shader code. It's not clear from your question which version of D3D you're using, but here's an overview of how to set up the blend stage in D3D11; other APIs are conceptually similar. The fields of the D3D11_RENDER_TARGET_BLEND_DESC1 structure let you specify how to combine the PS output color/alpha ("source") and the destination color/alpha ("dest") into the final color that gets written to the render target. One common setting is:

• SrcBlend = D3D11_BLEND_SRC_ALPHA
• DstBlend = D3D11_BLEND_INV_SRC_ALPHA
• BlendOp = D3D11_BLEND_OP_ADD

This uses the PS output alpha to blend directly between the render target color and the PS output color. If the output alpha is zero, the render target color is unchanged, and if the alpha is one, the render target color is totally replaced by the pixel shader output. Intermediate values blend smoothly between the two.

If you'd rather have strictly additive blending (where the pixel shader can only brighten the render target -- often used when rendering light quads, as you seem to be doing), then try:

• SrcBlend = D3D11_BLEND_SRC_ALPHA
• DstBlend = D3D11_BLEND_ONE
• BlendOp = D3D11_BLEND_OP_ADD

Many different effects are possible; check the docs for more details.

• I tried several forms of blending, but none of them do what I'm looking for. The only way I can get the effect I desire is if I could run the shader through the multiplied src and dest pixel. Aug 24 '12 at 11:15
• Can you express the effect you're looking for mathematically, in terms of a combination of the source and destination pixels? We might be able to suggest an appropriate set of blend parameters. The blend stage is surprisingly flexible, but if you really are trying to do something that can only be done in shader code, then your only remaining option is to bind the original render target as a second texture in the shader, do the blending in HLSL, and render to a new target. Aug 24 '12 at 16:09
• (src * dest) + shader_val... currently it is (src + shader_val) * dest. In the former, calculating shader_val takes into consideration the final pixel. Aug 24 '12 at 16:29
• The generalized alpha blend formula is "BlendOp(SrcColorSrcBlend, DstColorDstBlend)". SrcColor is the pixel shader output, DstColor is the original value of the destination pixel, and everything else are the values you provide. So, if you want to multiply SrcColor and DstColor, you could set SrcBlend=D3D11_BLEND_DEST_COLOR (or DstBlend=D3D11_BLEND_SRC_COLOR -- but not both!) and BlendOp=D3D11_BLEND_OP_ADD. I'm not sure where you're getting "shader_val" from, though -- can you elaborate? Aug 24 '12 at 17:38
• See how float4 c = (float4(attenuation,attenuation,attenuation,1.0) * color * float4(1.0,0.9,0.9,1.0) * 0.2) + (float4(attenuation2,attenuation2,attenuation2,1.0) * color * float4(1.0,0.8,1.0,1.0) * 0.2); So to make c, it multiplies the color into it. But what it would need would be this: float4 c = (float4(attenuation,attenuation,attenuation,1.0) * color * destColor * float4(1.0,0.9,0.9,1.0) * 0.2) + (float4(attenuation2,attenuation2,attenuation2,1.0) * color * destColor * float4(1.0,0.8,1.0,1.0) * 0.2); Aug 24 '12 at 18:01