Premultiplied Alpha And Alpha Testing

I have a shader that is supposed to work with either alpha blending or alpha testing, but the color values being passed in are premultiplied alpha values.

Is there an easy/standard way to have it produce the "correct" results for both alpha blending and alpha testing when using premultiplied alpha?

For example, if the final result of my shader is `RGBA(1,0,0,0.75)` straight alpha and therefore `RGBA(0.75,0,0,0.75)` premultiplied alpha, the result should be remain `RGBA(1,0,0,0.75)` when alpha testing.

Perhaps one option is "dividing out" the alpha channel if we are alpha testing, i.e. divide the `R` by `A` above so that we get `0.75/0.75 = 1.0`, our original red value. But this becomes non-deterministic when the alpha goes to 0.

Maybe I can clarify with an example:

Say the back buffer is `RGBA(1,1,1,1)`, and i am rendering with a partially transparent green color `RGBA(0,1,0,0.75)` (straight alpha). The color is passed in as premultiplied alpha as `RGBA(0,0.75,0,0.75)`. When alpha blending, i blend using one + invsrcalpha and get a final color of `RGBA(0.25,1,0.25,1)`. So my green channel stays at 100%, because the back buffer had green at 100%, and so did my original green color, so no way to get less than 100% on the green channel.

When Alpha testing, there is no blending going on. Instead, my green color will be rendered straight to the back buffer as `RGBA(0,0.75,0,1)`. But the color I wanted was `RGBA(0,1,0,1)`.

-
How does a different value in the R channel affect alpha testing? Alpha is the same in both cases. Also don't forget that you can use clip() in the shader to get the same effect as the render state. – Adam Mar 5 '13 at 19:48
Say I have a green tree alpha cutout. In the center, the tree might be `RGBA(0,1,0,1)`, but if its antialised around the edges of the cutout, some pixels might be `RGBA(0,0.75,0,0.75)` in premultiplied alpha. However, since i'm alpha testing and not alpha blending, I want the green color to stay 100% green and not get dark around the edges. Maybe there's a fundamental flaw in the way i'm imagining that alpha cutouts and anti-aliasing should work? – pauld Mar 5 '13 at 21:52
It shouldn't look dark if your alpha blending is set up right - you want the source blend factor to be one and not inverse source alpha. blogs.msdn.com/b/shawnhar/archive/2009/11/06/… explains the process well. – Adam Mar 5 '13 at 23:22
Right, I've read that article. But that applies to alpha blending. I know that alpha blending will work right and not look dark, but when alpha testing, the alpha channel written will be 100%. So I have a whole tree that is 100% green except around the edges it will be 75% green but 100% opaque. – pauld Mar 5 '13 at 23:52
You can't divide by A==0, but when A==0 you're not drawing anything anyway. So why does it matter? – Nevermind Mar 6 '13 at 4:44