Jagged transparency when rendering translucent textures (OpenGL 3.2 + GLSL)

Question

Like most others, I'll start off mentioning that I'm still a beginner when it comes to OpenGL and GLSL programming. So bear with me on any dumb mistakes you may spot in the code ahead.

I'm basically trying to render a model of a girl whose hair is implemented with transparent TGA textures. In my code I've setup OpenGL 3.2 (core profile, no fixed function calls) with the following calls before rendering the model's geometry (in clock-wise order):

glEnable(GL_DEPTH_TEST);
glEnable(GL_DEPTH_CLAMP);
glDepthMask(true);
glDepthFunc(GL_LEQUAL);
glDepthRange(0, 1);
glClearDepth(1.0);

glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);

In my fragment shader, I'm doing alpha premultiply of the textures coming in and I'm also discarding fragments with an alpha value lower than 0.5, like so:

vec4 SampledColor = texture2D(gColorMap, TexCoord0.xy);    

SampledColor.rgb *= SampledColor.a;

if (SampledColor.a < 0.5) {
    discard;
}

FragColor = SampledColor * TotalLight;
FragColor.a = SampledColor.a; 

The rendered image does display some transparency, but the model's hair looks jagged, here's a snapshot:

http://i32.photobucket.com/albums/d9/_Silencer/Samples/test_002.png

And here's the base texture being used to sample color values:

i32 . photobucket . com/albums/d9/_Silencer/Samples/sample.png

(note: photobucket only allowed me to upload sample as .png :P, but it's a TGA file).

I know for sure that the hard jaggies happen because I'm telling the shader to discard the fragment if falls below my arbitrary alpha threshold. What I'm wondering though is if there's any way to smoothly display the transparent pixels in the texture, pretty much like what can be seen on the right side of the rendered image, where the hair texture does blend in well with the gray background color.

I'm also pretty sure that I'm feeding an appropriate transparent image to OpenGL, but I can't find a way to get rid of the jagged artifacts.

Now, if I don't discard the fragment in the Fragment shader, the rendered hair texture will still appear transparent, but the "background" of the geometry adopts the background color of the color buffer, like so.

http://i32.photobucket.com/albums/d9/_Silencer/Samples/test_001.png

I've read that I may need to draw the geometry in sorted order according to the distance from the current viewpoint but I'm not sure how could I determine that with the hair chunks in this case.

So clearly I'm not understanding how to correctly handle translucent textures in rendered geometry. There has to be a nice way to do this since many games show this kind of textures correctly.

UPDATE The longer I continue investigating the subject, it seems that this is not a trivial problem. I may need to implement some sort of order independent transparency technique. I read that this can be done on OpenGL 4.0 capable hardware. What about OpenGL 3.3 hardware?

I appreciate any pointers or advice anyone could provide.

Thank you for your time and help!

Answer 1

I think the 0.5 falloff is really too strong and unnecessary; with 0.1 instead you will probably discard 95% of the fragments you already discard. You could even smooth the alpha to avoid remaining artifacts:

if (SampledColor.a < 0.1)
    discard;
SampledColor.rgb *= smoothstep(0.1, 1.0, SampledColor.a);

Answer 2

You have two options:

• Separate the hair geometry from the model and draw it after, ensuring a correct back-to-front order.
• Use alpha testing. You won't have nice blending this way but the hair contour will appear nicely sharp.