I'm trying to find a way to smooth out normals obtained from cross product of tangent and bitangent vectors calculated per face basis. I've already asked a similar question on SO, however with no answers so far.

Trying to come up with a solution (and also being bad at math) I decided to use the normals I obtain from the model file, since they are already given per vertex and transform each of the tangent plane vectors by rotation matrix obtained from dot and cross products of the calculated and loaded normals.

Now, this looks ok and the lighting is nice and smooth, however I have no idea if this solution is actually correct and, if no, what may be the problem with it. I assume it's not best as I haven't seen anything like it elsewhere.

Here's the part of the code which implements the idea. tangent bitangent and normal are the vectors computed per face. normal1 normal2 etc. are vertex normals.

    D3DXVECTOR3 axis;    // rotation axis   
    D3DXVECTOR4 temp;    // temporary vector for rotation result
    D3DXMATRIX rotation; // rotation matrix
    float cosAngle, angle;

    // Vertex 1
    // Find rotation axis and angle between face normal and vertex normal
    cosAngle = D3DXVec3Dot(&normal, &normal1);
    angle = acos(cosAngle);
    D3DXVec3Cross(&axis, &normal, &normal1);
    D3DXVec3Normalize(&axis, &axis);        

    // Create rotation matrix
    D3DXMatrixRotationAxis(&rotation, &axis, angle);

    // Transform each of the vectors
    D3DXVec3Transform(&temp, &normal, &rotation);
    normal1.x = temp.x;
    normal1.y = temp.y;
    normal1.z = temp.z;

    D3DXVec3Transform(&temp, &tangent, &rotation);
    tangent1.x = temp.x;
    tangent1.y = temp.y;
    tangent1.z = temp.z;

    D3DXVec3Transform(&temp, &bitangent, &rotation);
    bitangent1.x = temp.x;
    bitangent1.y = temp.y;
    bitangent1.z = temp.z;

    // do the same for vertices 2 and 3...
  • \$\begingroup\$ Your SO question is better than this one; this one is asking "is doing it this way OK" which is a yes/no question (and thus generally not considered a good one). I would edit your question, but in doing so it would become basically an exact cross-post of your SO question, and cross-posting questions within the SE network is not a good thing. Please delete your SO question if you would prefer to have this question here. \$\endgroup\$ – user1430 Feb 14 '14 at 17:08

You can calculate smooth tangent/bitangent vectors the same way that people usually do smooth normal vectors. That is, calculate them per face, then for each vertex, calculate the average of the vectors over all the faces that use that vertex.

This will work OK, but on some models you'll run into problems around UV seams, where the tangents and bitangents tend to change direction sharply. Generally speaking, wherever there is a UV discontinuity there will be a tangent/bitangent discontinuity as well.

| improve this answer | |
  • \$\begingroup\$ Yeah that sounds OK and it's something I've seen in other places, but I wonder: since the normals I obtain this way are going to be the same I already get from the model file, why not use them to adjust the normals I calculate per face and then use the same transformation for tangent and bitangent? Is there anything particularly wrong with the approach above? \$\endgroup\$ – Marian Feb 14 '14 at 14:24
  • \$\begingroup\$ @Marian For each vertex, you're still going to have a bunch of different tangents & bitangents, one coming from each face, after rotating them as you proposed. You'll still have to average or combine them somehow, so I don't really see the utility of doing the rotation business first. \$\endgroup\$ – Nathan Reed Feb 14 '14 at 18:01

Not the answer you're looking for? Browse other questions tagged or ask your own question.