MSAA deferred implementation issue

Question

I recently implemented MSAA in my deferred renderer, it looks good but I just got a feeling I might have done it wrong.

Here is what for example the directional light fragment shader looks like:

const float DEPTH_BIAS = 0.00005;                                                                                               \n \
                                                                                                                                \n \
layout(std140) uniform UnifDirLight                                                                                             \n \
{                                                                                                                               \n \
    mat4 mVPMatrix[4];                                                                                                          \n \
    mat4 mCamViewMatrix;                                                                                                        \n \
    vec4 mSplitDistance;                                                                                                        \n \
    vec4 mLightColor;                                                                                                           \n \
    vec4 mLightDir;                                                                                                             \n \
    vec4 mGamma;                                                                                                                \n \
    vec2 mScreenSize;                                                                                                           \n \
    int mNumSamples;                                                                                                            \n \
} UnifDirLightPass;                                                                                                             \n \
                                                                                                                                \n \
layout (binding = 2) uniform sampler2DMS unifPositionTexture;                                                                   \n \
layout (binding = 3) uniform sampler2DMS unifNormalTexture;                                                                     \n \
layout (binding = 4) uniform sampler2DMS unifDiffuseTexture;                                                                    \n \
layout (binding = 6) uniform sampler2DArrayShadow unifShadowmap;                                                                \n \
                                                                                                                                \n \
out vec4 fragColor;                                                                                                             \n \
                                                                                                                                \n \
void main()                                                                                                                     \n \
{                                                                                                                               \n \
    ivec2 texcoord = ivec2(textureSize(unifDiffuseTexture) * (gl_FragCoord.xy / UnifDirLightPass.mScreenSize));                 \n \
                                                                                                                                \n \
    vec3 worldPos = vec3(0.0), normal = vec3(0.0), diffuse = vec3(0.0);                                                         \n \
    for (int i = 0; i < UnifDirLightPass.mNumSamples; i++)                                                                      \n \
    {                                                                                                                           \n \
        worldPos += texelFetch(unifPositionTexture, texcoord, i).rgb;                                                           \n \
        normal   += texelFetch(unifNormalTexture, texcoord, i).rgb;                                                             \n \
        diffuse  += texelFetch(unifDiffuseTexture, texcoord, i).rgb;                                                            \n \
    }                                                                                                                           \n \
    worldPos /= UnifDirLightPass.mNumSamples;                                                                                   \n \
    normal   /= UnifDirLightPass.mNumSamples;                                                                                   \n \
    diffuse  /= UnifDirLightPass.mNumSamples;                                                                                   \n \
    normal = normalize(normal);                                                                                                 \n \
                                                                                                                                \n \
    vec4 camPos = UnifDirLightPass.mCamViewMatrix * vec4(worldPos, 1.0);                                                        \n \
                                                                                                                                \n \
    int index = 3;                                                                                                              \n \
    if (camPos.z > UnifDirLightPass.mSplitDistance.x)                                                                           \n \
        index = 0;                                                                                                              \n \
    else if (camPos.z > UnifDirLightPass.mSplitDistance.y)                                                                      \n \
        index = 1;                                                                                                              \n \
    else if (camPos.z > UnifDirLightPass.mSplitDistance.z)                                                                      \n \
        index = 2;                                                                                                              \n \
                                                                                                                                \n \
    vec4 projCoords = UnifDirLightPass.mVPMatrix[index] * vec4(worldPos, 1.0);                                                  \n \
    projCoords.w    = projCoords.z - DEPTH_BIAS;                                                                                \n \
    projCoords.z    = float(index);                                                                                             \n \
    float visibilty = texture(unifShadowmap, projCoords);                                                                       \n \
                                                                                                                                \n \
    float angleNormal = clamp(dot(normal, UnifDirLightPass.mLightDir.xyz), 0, 1);                                               \n \
                                                                                                                                \n \
    fragColor = vec4(diffuse, 1.0) * visibilty * angleNormal * UnifDirLightPass.mLightColor;                                    \n \
}                                                                                                                               \n";

1. I average the position/normals/diffuse before light and shadow calculations, but I just realised perhaps I should average the computed results instead. Which one is correct?

2. Right now I'm using hardware PCF for my shadows. Since I'm fetching several samples anyway for MSAA, couldn't I use that for shadow filtering aswell? Any downsides?

Answer 1

To be correct, lighting calculations should be carried out separately for each surface contained in a pixel, then the results should be averaged. For instance, think about the silhouette edge of an object, which contains some samples from the object and some other samples from whatever's behind it. If you simply averaged the positions, normals, etc. of all samples, you'd be lighting a surface that didn't exist, halfway between the object and the background.

One easy but slow way to do correct deferred lighting with MSAA is to run the lighting shaders at per-sample rate, which can be done with the ARB_sample_shading GL extension.

However, the majority of pixels on screen have only one visible surface in them and therefore don't need separate shading results for each subpixel sample. Some approaches to deferred MSAA try to detect which pixels contain multiple surfaces (by using heuristics based on depth, for instance) and perform per-sample shading only for those pixels - e.g. by marking them in the stencil buffer, and using stencil test to limit the per-sample shading. A simpler shader that only shades once per pixel can then be used for all other pixels on screen. Doing deferred MSAA efficiently is still an open problem that is being researched.