Anti Aliasing: How to implement supersampling in OpenGL?

Question

I'm learning OpenGL by working on a small Oculus Rift project, which implies, anti aliasing is everything :). Since the application of the barrel distortion shader requires rendering to a framebuffer in higher resolution anyways, my goal would be to perform super sampling anti aliasing (unfortunately, the very good answers to this question do not explain super sampling anti aliasing). In order to tackle the problem step by step, I first try to ignore the distortion aspect, and try to render my framebuffer directly.

The strange thing I realized is that, although I use a really huge resolution for the off-screen framebuffer (3840x2400; performance is not the issue), the outcome is not anti aliased at all.

I render my framebuffer by setting up a simple VBO quad in combination with the following vertex/fragment shaders:

Vertex Shader:

attribute vec2 Position;
attribute vec2 TexCoord;
varying vec2 oTexCoord;
void main()
{
  oTexCoord = TexCoord;
  gl_Position = vec4(Position, 0, 1);
}

Fragment Shader:

uniform sampler2D sampler;
varying vec2 oTexCoord;

void main()
{
  gl_FragColor = texture2D(sampler, oTexCoord);
}

I currently see four possible causes why my result is not anti aliased:

1. It's related to general texture rendering settings, which would go into the direction of this question, i.e., I have to change texture filtering or mipmapping (haven't learned these things yet). For sake of completeness: I'm using glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_LINEAR); for the texture attached to my FBO.

2. I have to manually implement a super sampling in the fragment shader, which would go into the direction of the comments in yet another question. The idea here would be that instead of using texture2D(sampler, oTexCoord) directly as fragment color, one would vary oTexCoord slightly left/right/up/down and blend the colors to a final result (similar to a Gaussian blur?). Is there some kind of standard fragment shader which accomplishes that in a sophisticated manner? I would actually be a bit surprised if this is really required, because this would imply that without extra work textures are never anti aliased by using a basic fragment shader?

3. I have to bind a multisampling texture to my FBO, which according to this tutorial has some issues that I do not fully understand. However, if I understand it correctly, what I'm looking for is not an anti aliasing in the off-screen FBO itself, but anti aliasing when rendering an (potentially arbitrary) texture to the primary framebuffer. Besides, the resolution of the FBO is already huge.

4. The last cause could be related to general OpenGL settings. In order to analyze this I factored out the whole rendering part allowing to switch between rendering via off-screen framebuffer and directly rendering to the primary framebuffer. I ensured that my display device (using LWJGL) is using 8 MSAA samples and I set:

   glEnable(GL_MULTISAMPLE_ARB)
   glHint(GL_MULTISAMPLE_FILTER_HINT_NV, GL_NICEST)
   

   Interestingly, I indeed get an anti-aliased result when I render directly! So the problem really seems to come from rendering my framebuffer texture.

Any hint is highly appreciated! I also would be glad to have a good reference/example on supersampling in OpenGL in general -- but, the fact that I have trouble finding anything tells me that I have have understood something completely wrong?

Update

Thanks to Fault's answer, hinting me towards linear texture interpolation, I'm now one step closer to solving the problem. The reason why I did not see a difference between GL_NEAREST and GL_LINEAR filtering was that I was simply using a framebuffer which was much too large! By switching from 8x supersampling to 2x supersampling I finally could see a difference.

I'll try to explain my findings for other OpenGL learners. Studying this great tutorial made clear why a too large framebuffer is a problem: Linear interpolation simply means that OpenGL will use the 4 closest texels from the texture to determine the fragment color. When the texture has a much higher texel density in comparison to the fragment area (see this illustration from the tutorial), this averaging simply has no big effect. The general solution for texturing is to use mipmaps, i.e., smaller size versions of the texture. For a framebuffer texture this is probably not a good idea (right?). Creating several mipmap levels for each rendered frame seems to be pretty inefficient.

My next question was why texture filtering does not simply determine the fragment area and iterate over the correct texels. The problem is that, in contrast to the FBO-to-screen-rendering, general texture filtering is not isotropic, i.e., the fragment area not necessarily lines up with the texel space (example).

So, if I see it correctly, there are two ways to implement supersampling in OpenGL:

1. By sticking to an oversampling factor of 2 and using the above shader with bilinear filtering. But imho the result looks worse in comparison to 4x MSAA (rendering without framebuffer). And one has to be aware of the counter-intuitive behavior that increasing the oversampling actually leads to worse quality.

2. Since for FBO-to-screen-rendering it is "rather simple" to compute the fragment area, a sophisticated aliasing fragment shader could be used. In comparison to the first approach this is probably much slower but should allow much better quality. Furthermore, increasing the size of the framebuffer now also (should) improve quality. However, I currently have no idea how to implement such an anti aliasing shader (how to calculate the fragment area? Should I use grid/stochastic/poisson disc sampling?). I still wasn't able to find any reference or example implementation.

Answer 1

glTexParameteri(GL_TEXTURE_2D, GL_TEXTURE_MAG_FILTER, GL_NEAREST);

GL_NEAREST tells the GPU to sample the nearest pixel, and the nearest pixel only. When downsampling, you want an average of the nearest pixels, so you should change it to GL_LINEAR. This tells it to linearly interpolate between the nearest pixels, thus if you sample at a point directly between 4 pixels, it will give you their average colour. Along an edge, it would sample the edge colour, and the background colour, and give you something in between (anti-aliasing).

Answer 2

As you have stated your approach works if the intermediate framebuffer is 2x the size of the final screen resolution. That is beause the bilinear sampling takes 4 texels into account. Now if your intermediate framebuffer is 4x the original screen size you will have to call texture2d four times with a certain offset, add the samples together and calculate the average by dividing by 4.

For best results you should also consider:

• make sure your sample points are exactly in the middle of 4 texels
• Implement bicubic sampling instead of using bilinear (you can switch back to Nearest for that)
• linearize your samples after fetch, gamma correct after averaging

For best performance:

• implement downsampling as a separable convolution along X and Y if multisampling required more than 5x5 samples