I've found a cheap solution in an article by Christian Schüler, where he posted a shader code used in Velvet Assassin. He smothes highlight by limiting maximum gloss to the surface curvature, which is computed by ddx and ddy of world-space normalized normals. It produces amazing results, maybe not so in noice reduction but in temporal stability of highlights (fancy graphical term, which means how much something changes across frames). Screenshots will not show the effect well enough, so I will not post them, but in action I got rid of all specular flickering on moving objects.
For those of you who are lazy (like me :)) and don't want to scan the code for the needed parts (I highly suggest that you do though, because the shader is stellar for learning purpouses) here are the relevant bits:
Add this after you transformed normal into world space and normalized it:
float3 ddxN = ddx(Normal.rgb);
float3 ddyN = ddy(Normal.rgb);
float curv2 = max( dot( ddxN, ddxN ), dot( ddyN, ddyN ) );
float gloss_max = -0.0909h - 0.0909h * log2(CURV_MOD*curv2);
gloss = min(gloss, gloss_max);
gloss = min(MAX_POW, exp2(1 + lerp(POW_MOD_MIN, POW_MOD_MAX, gloss )));
Compute specular this way:
half D = (gloss + 1) * pow(NdotH, gloss);
specularColor = lightColor.rgb * NdotL * D;
CURV_MOD
- how much smothing to apply. Be careful it can make some small objects too shiny. I have it set to 2.
MAX_POW
- the maximum specular power you want to have in your engine. I have it set to 2000.
POW_MOD_MIN
, POW_MOD_MAX
- gloss boundaries. Change the lower value if you want very dull but super broad specular at low gloss values, change the higher value if you want extremely sharp specular with low AA at high gloss values. Top value will be limited by MAX_POW as well. My values are 5, 15.
Experiment! I had to multiply the final specular result by 1/100 to make it consistent with my lighting, but for the rest I pretty seamlessly transitioned from using just a simple lit() function for specular.
Yeah one more thing - I had to change my specular intensity depending on gloss - the lower the gloss - the higher the intensity. Otherwise my highlights were too dull at low gloss.
Once again all credit for this solution goes to Christian Schüler.