How do I implement anti-aliasing in OpenGL?

Question

I want to do full-screen anti-aliasing in OpenGL, and dont want to use the anti-aliasing that is provided by OpenGL itself, since I am building a game and want good effects in it.

How should I proceed?

Answer 1

There are several alternatives to native MSAA in OpenGL. With post-processing effects, the best thing about them is that you can usually just throw in the shader to the final, unprocessed image and it does the rest. Here are three methods worth taking a look:

• Fast Approximate Anti-Aliasing (Geeks3D) - Good in most cases. Pretty easy to apply and understand. Drawback is sharp, high contrast noise in textures gets blurred a bit. Edges as subtle as 1/4 pixels steep look dead-accurate as traditional MSAA. Any less steep than that, it loses a bit of accuracy.

• Normal Filtered Anti-Aliasing (GameDev) - Haven't tested this one yet accurately, but it's the easiest to understand. In best cases it resembles 16x MSAA and in worst cases it's like 2x MSAA. It generates a temporary normal map to represent edges and relative angles. You can sample the normal map either with luma difference or color difference.

• Morphological Anti-Aliasing (Iryoku) - been improved to SMAA - Subpixel Mophological AA. It's pretty complex at 4 passes, but achieves the best results I've seen. It creates gradients along edges as gradual as 1/100 to 1/200 pixels steep (!). Sampling can be luma-based, color-based or depth-based. Textures stay very crisp and clean. (the example is DX10 and HLSL based, would take some time to port it to GLSL accurately)

These techniques don't super-sample or multi-sample, so lines that appear less than 1 pixel in thickness will appear with gaps and not be anti-aliased correctly. This is the downside to using a non-MSAA approach. Since you're only working with a raster image at full resolution, you can't create additional information from these empty gaps.

Take notice that all of these techniques are dependent on sampling adjacent luma (brightness) or chroma (color) values. Calculating luma and optional gamma correction requires additional instructions on the AA shader, though it's pretty straightforward. You can offload this by calculating the luma in the previous shader that provides the un-retouched image, storing the luma in the alpha channel. Then in the AA shader, you will simply sample the alpha.

Answer 2

There's lots of ways to do antialiasing. One is to use multisample antialiasing (MSAA), where your back buffer actually stores multiple sub-pixel samples, and when you render triangles, lines, etc. the system automatically fills in the correct set of samples in each pixel. Then at the end of rendering the image is "resolved" by averaging over all the sub-pixel samples to get one sample per pixel.

Another way is to use post-processing antialiasing, where you render the scene as normal and then do some targeted blurring on the end result to hide aliased edges. There are a variety of techniques for this, but one of the best / most popular at the moment is called FXAA (Fast approXimate Anti-Aliasing).

MSAA will generally give better-looking results than post-processing AA, but can be slower because it requires approximately double the memory bandwidth for every rendering operation. MSAA can also require more video memory than post-processing AA, depending on the details of the setup.

You can find specific info about implementing either MSAA or FXAA on the Web - just google either one of those terms.