Dealing with alpha for 2D per pixel lighting

Question

Right now I've got a simple light shader. Every bitmap I draw goes through it to make up the scene. I only draw quads.

   texture tex;
   sampler2D s = sampler_state {
      texture = <tex>;
   };

bool use_tex;

float x = 0.1;
float y = 0.3;

   float4 ps_main(VS_OUTPUT Input) : COLOR0
   {
float4 p = Input.vPosition / Input.vPosition.w;
      float4 color = 0;
      float a = color.a;
      if(!use_tex)
        {
           color = Input.Color;
        }
        else
        {
           color = Input.Color * tex2D(s, Input.TexCoord.xy); 
        }
      p += 1.0;
      p /= 2;
      p.y = 1.0 - p.y;


      float3 lightAttenuation = float3(0.05,4.0, 0.0);
     // float4 color = Input.Color * tex2D(s, Input.TexCoord.xy);
      float2 aux = float2(x,y) - p.xy;
      float dist = length(aux);
      float attenuation=1.0/(lightAttenuation.x+lightAttenuation.y*dist

+lightAttenuation.z*dist*dist);



      float3 lightAttenuation2 = float3(0.05,3.0, 0.0);
      float2 aux2 = float2(0.5,0.4) - p.xy;
      float dist2 = length(aux2);
      float attenuation2=1.0/(lightAttenuation2.x

+lightAttenuation2.y*dist2+lightAttenuation2.z*dist2*dist2);


      color = (float4(attenuation,attenuation,attenuation,1.0) * color * float4(1.0,0.8,0.8,1.0) 

* 0.8)  + (float4(attenuation2,attenuation2,attenuation2,1.0) * color * float4(1.0,0.8,1.0,1.0) 

* 0.2);




      return color;
   }

The shader calculates based on the current pixel's screen position.

The problem is when the pixel's alpha is != 1.0

It this case it would need to be multiplied by what is already in the backbuffer at that pixel location in order to get the correct final pixel.

What can I do to fix the alpha issue and deal with the alpha case?

Thanks

PS2.0 hlsl.

Edit

Here are 2 images, the bright one is when the whole scene is rendered to texture then the shader is applied. The second is when I apply the shader on each fragment as the scene is rendered. This is an area where there is a dark shadow on the ground. The shadow is a semi transparent bitmap. Thus it makes the lighting effect weaker and it makes it look weird.

What I want is a way for it to look like the first one, but still letting me do per fragment shading. What I was saying was that I thought of maybe looking at what is already in the backbuffer, then multiplying it by the current pixel, then taking that result, applying the shader to it, then obviously you set the alpha to 1.0.

Im not sure if or how that could work. Im wondering either how to do that, or a better way to get the same result.

Here is my shader source:

   texture tex;
   sampler2D s = sampler_state {
      texture = <tex>;
   };

bool use_tex;

float x = 0.1;
float y = 0.3;

   float4 ps_main(VS_OUTPUT Input) : COLOR0
   {
float4 p = Input.vPosition / Input.vPosition.w;
      float4 color = 0;

      if(!use_tex)
        {
           color = Input.Color;
        }
        else
        {
           color = Input.Color * tex2D(s, Input.TexCoord.xy); 
        }
     float a = color.a;
      p += 1.0;
      p /= 2;
      p.y = 1.0 - p.y;


      float3 lightAttenuation = float3(0.05,4.0, 0.0);
     // float4 color = Input.Color * tex2D(s, Input.TexCoord.xy);
      float2 aux = float2(x,y) - p.xy;
      float dist = length(aux);
      float attenuation=1.0/(lightAttenuation.x+lightAttenuation.y*dist+lightAttenuation.z*dist*dist);



      float3 lightAttenuation2 = float3(0.0,2.1, 1.0);
      float2 aux2 = float2(-1.5 + x,0.4 + y) - p.xy;
      float dist2 = length(aux2);
      float attenuation2=1.0/(lightAttenuation2.x+lightAttenuation2.y*dist2+lightAttenuation2.z*dist2*dist2);


      float4 c = (float4(attenuation,attenuation,attenuation,1.0) * color * float4(1.0,0.9,0.9,1.0) * 0.2) + (float4(attenuation2,attenuation2,attenuation2,1.0) * color * float4(1.0,0.8,1.0,1.0) * 0.2);

      color = c;


      color.a = a;

      return color;
   }

Answer 1

You're right, pixel shaders don't let you read from the destination pixel in the render target. That's what the alpha blending stage is for: combining the pixel shader output color with the current color in the destination render target.

The blend stage is configured by the application, not the shader code. It's not clear from your question which version of D3D you're using, but here's an overview of how to set up the blend stage in D3D11; other APIs are conceptually similar. The fields of the D3D11_RENDER_TARGET_BLEND_DESC1 structure let you specify how to combine the PS output color/alpha ("source") and the destination color/alpha ("dest") into the final color that gets written to the render target. One common setting is:

• SrcBlend = D3D11_BLEND_SRC_ALPHA
• DstBlend = D3D11_BLEND_INV_SRC_ALPHA
• BlendOp = D3D11_BLEND_OP_ADD

This uses the PS output alpha to blend directly between the render target color and the PS output color. If the output alpha is zero, the render target color is unchanged, and if the alpha is one, the render target color is totally replaced by the pixel shader output. Intermediate values blend smoothly between the two.

If you'd rather have strictly additive blending (where the pixel shader can only brighten the render target -- often used when rendering light quads, as you seem to be doing), then try:

• SrcBlend = D3D11_BLEND_SRC_ALPHA
• DstBlend = D3D11_BLEND_ONE
• BlendOp = D3D11_BLEND_OP_ADD

Many different effects are possible; check the docs for more details.