I want to do full-screen anti-aliasing in OpenGL, and dont want to use the anti-aliasing that is provided by OpenGL itself, since I am building a game and want good effects in it.
How should I proceed?
There are several alternatives to native MSAA in OpenGL. With post-processing effects, the best thing about them is that you can usually just throw in the shader to the final, unprocessed image and it does the rest. Here are three methods worth taking a look:
These techniques don't super-sample or multi-sample, so lines that appear less than 1 pixel in thickness will appear with gaps and not be anti-aliased correctly. This is the downside to using a non-MSAA approach. Since you're only working with a raster image at full resolution, you can't create additional information from these empty gaps.
Take notice that all of these techniques are dependent on sampling adjacent luma (brightness) or chroma (color) values. Calculating luma and optional gamma correction requires additional instructions on the AA shader, though it's pretty straightforward. You can offload this by calculating the luma in the previous shader that provides the un-retouched image, storing the luma in the alpha channel. Then in the AA shader, you will simply sample the alpha.
There's lots of ways to do antialiasing. One is to use multisample antialiasing (MSAA), where your back buffer actually stores multiple sub-pixel samples, and when you render triangles, lines, etc. the system automatically fills in the correct set of samples in each pixel. Then at the end of rendering the image is "resolved" by averaging over all the sub-pixel samples to get one sample per pixel.
Another way is to use post-processing antialiasing, where you render the scene as normal and then do some targeted blurring on the end result to hide aliased edges. There are a variety of techniques for this, but one of the best / most popular at the moment is called FXAA (Fast approXimate Anti-Aliasing).
MSAA will generally give better-looking results than post-processing AA, but can be slower because it requires approximately double the memory bandwidth for every rendering operation. MSAA can also require more video memory than post-processing AA, depending on the details of the setup.
You can find specific info about implementing either MSAA or FXAA on the Web - just google either one of those terms.