I'm trying to find a way to smooth out normals obtained from cross product of tangent and bitangent vectors calculated per face basis. I've already asked a similar question on SO, however with no answers so far.
Trying to come up with a solution (and also being bad at math) I decided to use the normals I obtain from the model file, since they are already given per vertex and transform each of the tangent plane vectors by rotation matrix obtained from dot and cross products of the calculated and loaded normals.
Now, this looks ok and the lighting is nice and smooth, however I have no idea if this solution is actually correct and, if no, what may be the problem with it. I assume it's not best as I haven't seen anything like it elsewhere.
Here's the part of the code which implements the idea.
normal are the vectors computed per face.
normal2 etc. are vertex normals.
D3DXVECTOR3 axis; // rotation axis D3DXVECTOR4 temp; // temporary vector for rotation result D3DXMATRIX rotation; // rotation matrix float cosAngle, angle; // Vertex 1 // Find rotation axis and angle between face normal and vertex normal cosAngle = D3DXVec3Dot(&normal, &normal1); angle = acos(cosAngle); D3DXVec3Cross(&axis, &normal, &normal1); D3DXVec3Normalize(&axis, &axis); // Create rotation matrix D3DXMatrixRotationAxis(&rotation, &axis, angle); // Transform each of the vectors D3DXVec3Transform(&temp, &normal, &rotation); normal1.x = temp.x; normal1.y = temp.y; normal1.z = temp.z; D3DXVec3Transform(&temp, &tangent, &rotation); tangent1.x = temp.x; tangent1.y = temp.y; tangent1.z = temp.z; D3DXVec3Transform(&temp, &bitangent, &rotation); bitangent1.x = temp.x; bitangent1.y = temp.y; bitangent1.z = temp.z; // do the same for vertices 2 and 3...