I've been trying to replicate the result of this paper (linked directly to PDF paper, but you don't need to read it to give an answer) and encountered a problem with the BRDF according to the following well-known formula:
where F,G,D are Fresnel reflection, Shadow masking, and microfacet distribution respectively. The trouble I encounter is in the denominator, where i is the incident ray, n is the surface normal, and o is the angle of view.
Just by looking at the formula, probably you can already guess what happen when the ray comes from grazing angle: |in| approach zero, and if I look at it also at grazing angle then |on| would also approach zero. And as much as I hoped that FGD would approach zero faster than the denominator, it doesn't. (I used GGX distribution, but the same thing happen to Beckmann or Phong) The resulting BRDF I calculated is something completely different from not only the paper that I'm trying to replicate, but also from most other BRDF.
(strong peak at grazing angle, I did it for refraction case but it's the same for reflection)
(result from the paper, most brdf have this shape)
I double checked D,F and G so it can't be any of those, but then what went wrong? Has anyone tried this BRDF model and encountered a similar problem?