I'm trying to find a way to smooth out normals obtained from cross product of tangent and bitangent vectors calculated per face basis. I've already asked a similar question on SO, however with no answers so far.
Trying to come up with a solution (and also being bad at math) I decided to use the normals I obtain from the model file, since they are already given per vertex and transform each of the tangent plane vectors by rotation matrix obtained from dot and cross products of the calculated and loaded normals.
Now, this looks ok and the lighting is nice and smooth, however I have no idea if this solution is actually correct and, if no, what may be the problem with it. I assume it's not best as I haven't seen anything like it elsewhere.
Here's the part of the code which implements the idea. tangent
bitangent
and normal
are the vectors computed per face. normal1
normal2
etc. are vertex normals.
D3DXVECTOR3 axis; // rotation axis
D3DXVECTOR4 temp; // temporary vector for rotation result
D3DXMATRIX rotation; // rotation matrix
float cosAngle, angle;
// Vertex 1
// Find rotation axis and angle between face normal and vertex normal
cosAngle = D3DXVec3Dot(&normal, &normal1);
angle = acos(cosAngle);
D3DXVec3Cross(&axis, &normal, &normal1);
D3DXVec3Normalize(&axis, &axis);
// Create rotation matrix
D3DXMatrixRotationAxis(&rotation, &axis, angle);
// Transform each of the vectors
D3DXVec3Transform(&temp, &normal, &rotation);
normal1.x = temp.x;
normal1.y = temp.y;
normal1.z = temp.z;
D3DXVec3Transform(&temp, &tangent, &rotation);
tangent1.x = temp.x;
tangent1.y = temp.y;
tangent1.z = temp.z;
D3DXVec3Transform(&temp, &bitangent, &rotation);
bitangent1.x = temp.x;
bitangent1.y = temp.y;
bitangent1.z = temp.z;
// do the same for vertices 2 and 3...