A position v and normal n are given in world coordinates. Also there is a view matrix V for world-eye coordinate transition and a projection matrix P for eye-ndc transition. I would like to determine, whether the normal is facing the viewer.
If we assume that P is a perspective matrix, we can compare the normal direction with the direction to the viewer, both in eye-space. Note that the approach won't work for an orthographic projection, because there is no well defined eye position. But if P is orthographic, we only need to check the Z coordinate of n in eye space. Nevertheless, I'm seeking a solution that does not make assumption about what projection type we're using.
One way of achieving this is to compute the inverse P' of P and retrieve the viewing direction as a vector spanned between P' [u.x, u.y, -1] and P' [u.x, u.y, 1], where u is v transformed to ndc. The drawback here is that it requires inverting a matrix and in my case also passing it to the shader.
Another solution would be to construct a small triangle around v within the normal plane of n, and check what's its winding direction in the ndc. The problem with this approach is that it's computationally heavy and it's still seems to me more like a hack than a proper solution.
I am not very satisfied with these two solutions. Do you have any other ideas? Am I missing something very simple?
EDIT:
Since it seems unclear why the dot(v - eye, n) < 0 is not satisfactory in my case, I post an exaple showing the difference between perspective and ortho projection for this particular problem.
As you can see, the vector is facing slightly away in the perspective projection, but clearly towards in the orthographic projection. More specifically, a face having that particular normal would be culled in perspective, but not in ortho.
dot(eye-v, n)>0\$\endgroup\$