Jagged transparency when rendering translucent textures (OpenGL 3.2 + GLSL)

Like most others, I'll start off mentioning that I'm still a beginner when it comes to OpenGL and GLSL programming. So bear with me on any dumb mistakes you may spot in the code ahead.

I'm basically trying to render a model of a girl whose hair is implemented with transparent TGA textures. In my code I've setup OpenGL 3.2 (core profile, no fixed function calls) with the following calls before rendering the model's geometry (in clock-wise order):

glEnable(GL_DEPTH_TEST);
glEnable(GL_DEPTH_CLAMP);
glDepthFunc(GL_LEQUAL);
glDepthRange(0, 1);
glClearDepth(1.0);

glEnable(GL_BLEND);
glBlendFunc(GL_SRC_ALPHA, GL_ONE_MINUS_SRC_ALPHA);


In my fragment shader, I'm doing alpha premultiply of the textures coming in and I'm also discarding fragments with an alpha value lower than 0.5, like so:

vec4 SampledColor = texture2D(gColorMap, TexCoord0.xy);

SampledColor.rgb *= SampledColor.a;

if (SampledColor.a < 0.5) {
}

FragColor = SampledColor * TotalLight;
FragColor.a = SampledColor.a;


The rendered image does display some transparency, but the model's hair looks jagged, here's a snapshot:

http://i32.photobucket.com/albums/d9/_Silencer/Samples/test_002.png

And here's the base texture being used to sample color values:

i32 . photobucket . com/albums/d9/_Silencer/Samples/sample.png


(note: photobucket only allowed me to upload sample as .png :P, but it's a TGA file).

I know for sure that the hard jaggies happen because I'm telling the shader to discard the fragment if falls below my arbitrary alpha threshold. What I'm wondering though is if there's any way to smoothly display the transparent pixels in the texture, pretty much like what can be seen on the right side of the rendered image, where the hair texture does blend in well with the gray background color.

I'm also pretty sure that I'm feeding an appropriate transparent image to OpenGL, but I can't find a way to get rid of the jagged artifacts.

Now, if I don't discard the fragment in the Fragment shader, the rendered hair texture will still appear transparent, but the "background" of the geometry adopts the background color of the color buffer, like so.

http://i32.photobucket.com/albums/d9/_Silencer/Samples/test_001.png

I've read that I may need to draw the geometry in sorted order according to the distance from the current viewpoint but I'm not sure how could I determine that with the hair chunks in this case.

So clearly I'm not understanding how to correctly handle translucent textures in rendered geometry. There has to be a nice way to do this since many games show this kind of textures correctly.

UPDATE The longer I continue investigating the subject, it seems that this is not a trivial problem. I may need to implement some sort of order independent transparency technique. I read that this can be done on OpenGL 4.0 capable hardware. What about OpenGL 3.3 hardware?

I appreciate any pointers or advice anyone could provide.

Thank you for your time and help!

• Are you for sure rendering the hair after all solid objects? Are you depth-sorting the hair? Translucency requires strict painter's algorithm ordering to rendering, or high-powered techniques like depth peeling. The fragments behind the translucent pixel must be rendered first or it has nothing to blend with. – Sean Middleditch Feb 12 '13 at 0:38
• Yeah I thought about all that as well. Currently I'm not sorting the geometry. I could thought since I know which indices form each triangle in the mesh. However, I've been reading that Order Independent Transparency works much more reliably than geometry sorting, so I'm going to investigate if that technique can be implemented in OpenGL 3.3 capable hardware. Thanks! – Jesús Zazueta Feb 13 '13 at 23:30
• Order Independent Transparency is a catch-all for multiple techniques. Some are complex, all have catches. Sorting is easy and relatively fool-proof so long as any complex overlapping geometry is broken up to not longer overlap, which may be difficult with hair. Another trick is to just model the hair so you can render it last. You can ignore most hair artifacts from self-occlusion issues, they are hard to notice. – Sean Middleditch Feb 14 '13 at 7:30

I think the 0.5 falloff is really too strong and unnecessary; with 0.1 instead you will probably discard 95% of the fragments you already discard. You could even smooth the alpha to avoid remaining artifacts:

if (SampledColor.a < 0.1)
SampledColor.rgb *= smoothstep(0.1, 1.0, SampledColor.a);

• Thanks for the tip! So if I understand correctly, your code suggest to discard the fragment early if it does not pass the lowered alpha threshold test. What I'm not sure is about the next line. Reading the documentation for smoothstep it seems to be performing Hermite interpolation; in this case does it say the following? "The sampled color's RGB value shall be the same value, multiplied by the Hermite interpolation of the sample's alpha value, ranging from 0.1 to 1.0" Is that correct? – Jesús Zazueta Feb 11 '13 at 19:24
• ... in other words, what is smoothstep doing in this case? My apologies, I'm still trying to understand how the Fragment Shader is working. Thanks! – Jesús Zazueta Feb 11 '13 at 19:41
• smoothstep may not be necessary at all; but its purpose here is to avoid any visible discontinuities when the alpha is near 0.1, by stretching these values to 0.0 using a smooth curve. – sam hocevar Feb 11 '13 at 21:12
• Maybe I did something wrong. Adding smoothstep to the Fragment Shader code made the model look completely dark XD. i32.photobucket.com/albums/d9/_Silencer/Samples/test_003.png Should't I use the function like this instead? Thanks! SampledColor.rgb *= smoothstep(0.1, 1.0, SampledColor.a);  – Jesús Zazueta Feb 11 '13 at 23:07
• You are right. I used the wrong argument order. Thanks, answer fixed! – sam hocevar Feb 12 '13 at 0:00

You have two options:

• Separate the hair geometry from the model and draw it after, ensuring a correct back-to-front order.
• Use alpha testing. You won't have nice blending this way but the hair contour will appear nicely sharp.
• Hi. Thanks. I guess option one may be a bit difficult as the model is a single OBJ mesh. I'd have to do some clever triangle sorting I guess. As for number two, is alpha testing something I would have to enable via GL commands, or something I could do in the fragment shader as well? Thanks again! – Jesús Zazueta Feb 11 '13 at 19:20
• Alpha testing is what you are already doing in your shader with the discard instruction. Somehow I misread your question and didn't see it when answering! – r2d2rigo Feb 11 '13 at 20:57