How can I reduce aliasing in my outline glow effect?

Question

I'm trying to replicate the glowing outline effect in the Left 4 Dead game. The effect causes an objects outline to glow, even when the object is occluded. Here is a screenshot of the effect:

I'm somewhat able to replicate this effect in my OpenGL based program. This is what I'm currently doing:

• Create a color and depth texture that is half the screen size for rendering the glowing objects
• Clear the glow color/depth textures. Color is cleared to black.
• For each glowing object, render it to the glow texture as a solid color
• Perform a separable gaussian blur on the glow texture
• Render the full resolution scene as normal
• Additively blend the glow texture with the normal scene, but use the glow depth texture to mask out the object, leaving just the blurred outline.

Here is a screenshot of my approach:

Here is the fragment shader that combines the glow texture with the scene:

uniform sampler2D glowColorTex;
uniform sampler2D glowDepthTex;
uniform sampler2D sceneColorTex;
void main()
{
    vec2 uv = gl_TexCoord[0].st;

    vec4 color = texture2D( sceneColorTex, uv);

    float depth = texture2D( glowDepthTex, uv).r;
    if(depth == 1.0) {
        color += 2.0 * texture2D( glowColorTex, uv);
    }

    gl_FragColor = color;
}

As you can see, it seems to work for the most part, but the aliasing on the outline is really bad.

Does anybody have any suggestions for smoothing the inner edge of the outline?

Should I be sampling the neighboring depth values of each pixel and scale the glow based on the number of depth values that equal 1.0?

Or is there a better approach that will produce smoother results?

Answer 1

Maybe instead of checking it against the low-resolution depth buffer you might use the stencil test. When rendering the normal scene, you just render the object that should glow into the stencil buffer (without depth-testing, I think) and then blend the complete glow texture in, but configure the stencil-test to only pass where the stencil buffer is not set, therefore masking out the high-resolution object.

This way you get the exact silhouette of the original object, whereas a smoothing of the edges would only lead to approximate results, but maybe these would suffice for you.