XNA 4.0 - Strange edges with multilight shader

Question

I am generating a light, a depth and a normalmap to calculate the lightning at each pixel with multiple lights.

On both rendertargets, i set the preferredMultiSampleCount parameter to 16 samples because I want smooth edges. It seems to work well on the first picture (normalmap). In the second picture (lightmap), multisampling works great, too. But somehow there are flickering edges at the end of each model.

I think it may have to do something with the depthmap because due to using Surfaceformat.Single I have to use

minfilter = point;
magfilter = point;
mipfilter = point;

for the Texture filter and I think AA doesnt work with filter type point.

Is this a known newbie problem or should I upload the source of my shader files for a better understanding of my problem?

EDIT: If I turn multisampling off, I dont get these edges: (Take a look at the bottom of the square in the middle.. )

Answer 1

If you're using any form of deferred lighting/shading, you'll have problems with multisample antialiasing because pixel shaders normally only run once per pixel. When you do your lighting passes, the depth and normal values within each pixel will get averaged out before being seen by the shader. This will cause problems at edges, where you have very different depth and normals at different samples, so the lighting result will be totally wrong.

Ideally, at edges the lighting would be calculated once for each surface present in the pixel. Unfortunately, in DX9 (which XNA is built on) it is not possible to access the individual samples in a multisample buffer, so as far as I'm aware there is no way to do deferred shading correctly with multisampling. (In DX11, it is possible.)

As János mentioned in the comments, an alternative approach is postprocess antialiasing, where you render the scene normally (without MSAA) and apply a postprocess filter such as FXAA, MLAA, or SMAA, which attempt to detect jaggies and blur them out. This can work quite well, though it's ultimately limited by the resolution of the input image.

Supersample antialasing is another alternative, though quite expensive. This would involve rendering the scene at 2x (or more) of the final resolution), doing all lighting and shading at this high resolution, then scaling down to the final resolution at the end.