0
\$\begingroup\$

I need help understanding what am I doing wrong with transformation/calculation of tangents and binormals using GLSL. I'm using Willems' PBR demo shaders as a reference: https://github.com/SaschaWillems/Vulkan-glTF-PBR/blob/master/data/shaders/pbr.vert https://github.com/SaschaWillems/Vulkan-glTF-PBR/blob/master/data/shaders/pbr_khr.frag

I start by programmatically generating a sphere with normals and texture coordinates as seen in this image: enter image description here

The "light" is directly to the left (-10, 0, 0) and both texture coordinates and normals seem to be proper as per smooth gradient and appropriately wrapped texture

However the moment I try to add binormal and tangent into the equation using the "getNormals" code from fragment shader above - it all gets messed up, lit from improper direction, not evenly and blocky: enter image description here

I tried to RenderDoc that PBR demo and the only big difference I see thus far is that he uses right handed system (Z is going backwards) whereas I'm using left-handed system (Z is forward) through GLM - but I'm not sure if that's what ruins calculations?

Do note that I'm using identity matrices so the only transformation that sphere encounters is projection*view as vertex main output and identity as "world position" (worldPos) output because there are no world-level transformations happening to vertices themselves.

I then feed this worldPos together with texCoord and normal into getNormals() using the following code:

vec3 Q1 = dFdx(worldPos);
vec3 Q2 = dFdy(worldPos);
vec2 st1 = dFdx(texCoord);
vec2 st2 = dFdy(texCoord);

vec3 N = normalize(normal);
vec3 T = normalize(Q1 * st2.t - Q2 * st1.t);
vec3 B = -normalize(cross(N, T));

mat3 TBN = mat3(T, B, N);

return normalize(TBN * normal); // transform normal

and get that issue. Setting T and B to vec3(0.0) both returns the smooth shading back and the result is equal to just passing normal as return.

does it have anything to do with the coordinate system I'm using? E.g. X is right, Y is up, Z is forward, U is right, V is down?

any ideas or tips?

\$\endgroup\$
4
  • \$\begingroup\$ Why are you multiplying the same normal used to calculate the TBN matrix by that matrix? This is not a geometrically meaningful operation. A TBN matrix is useful for transforming a tangent space normal read from a normal map texture into world space, but as shown here, the variable normal is treated as though it's already in world space. \$\endgroup\$
    – DMGregory
    Commented Mar 10, 2023 at 17:38
  • \$\begingroup\$ Figures, but when I'm reading normal from normal map texture - I get another weird result where triangles are improperly shaded and are visible \$\endgroup\$
    – krz
    Commented Mar 10, 2023 at 17:49
  • \$\begingroup\$ That's coming from the fact that you're computing your tangent vectors per fragment - so they're using partial derivatives that are constant over each triangle but discontinuous at triangle edges - instead of computing them per vertex and sending them to the GPU as vertex attributes, so all adjacent triangles agree about the tangents along their shared edges. \$\endgroup\$
    – DMGregory
    Commented Mar 10, 2023 at 18:17
  • \$\begingroup\$ but what I don't understand is why glTF-PBR demo that does the same - has smooth triangle shading whereas my attempts at doing the same result in visible triangle edges \$\endgroup\$
    – krz
    Commented Mar 10, 2023 at 18:24

1 Answer 1

0
\$\begingroup\$

I resolved it a long time ago, just wanted to post an answer. I was suffering by trying to "fix" the shader for an unexpected reason - an incorrect compression of a normal map using NVidia Texture Tools. It was messing up all the tangents causing a ton of graphical issues. Upon selecting a fitting compression the issue was resolved.

\$\endgroup\$
2
  • 1
    \$\begingroup\$ This answer would be even more helpful to future readers if you document what compression type was causing the problem, and what compression you switched to that fixed it. \$\endgroup\$
    – DMGregory
    Commented Dec 12, 2023 at 13:04
  • \$\begingroup\$ it was a long time ago but from what I remember the issue was with me saving it as a 'Image Type: Normal Map' in NVidia Texture Tools. Upon reopening the converted texture later I found out that what NTT was doing is converting an already existing, proper normal map into a normal map as if the source was a "diffuse" texture. Naturally when calculating light vector in the shader all normals were all over the place when rendering. Saving it as a typical "Color Map" resolved the issue. \$\endgroup\$
    – krz
    Commented Dec 21, 2023 at 20:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .