I have a shader that is supposed to work with either alpha blending or alpha testing, but the color values being passed in are premultiplied alpha values.
Is there an easy/standard way to have it produce the "correct" results for both alpha blending and alpha testing when using premultiplied alpha?
For example, if the final result of my shader is
RGBA(1,0,0,0.75) straight alpha and therefore
RGBA(0.75,0,0,0.75) premultiplied alpha, the result should be remain
RGBA(1,0,0,0.75) when alpha testing.
Perhaps one option is "dividing out" the alpha channel if we are alpha testing, i.e. divide the
A above so that we get
0.75/0.75 = 1.0, our original red value. But this becomes non-deterministic when the alpha goes to 0.
Maybe I can clarify with an example:
Say the back buffer is
RGBA(1,1,1,1), and i am rendering with a partially transparent green color
RGBA(0,1,0,0.75) (straight alpha). The color is passed in as premultiplied alpha as
RGBA(0,0.75,0,0.75). When alpha blending, i blend using one + invsrcalpha and get a final color of
RGBA(0.25,1,0.25,1). So my green channel stays at 100%, because the back buffer had green at 100%, and so did my original green color, so no way to get less than 100% on the green channel.
When Alpha testing, there is no blending going on. Instead, my green color will be rendered straight to the back buffer as
RGBA(0,0.75,0,1). But the color I wanted was