I'm getting different signs when I convert a matrix to quaternion and invert that, versus when I invert a matrix and then get the quaternion from it:
Quaternion a = Quaternion.Invert(getRotation(m)); Quaternion b = getRotation(Matrix.Invert(m));
I would expect a and b to be identical (or inverses of each other). However, it looks like q1 = (x, y, -z, -w) while q2 = (-x, -y, w, z). In other words, the Z and W components have been switched for some reason.
getRotation() decomposes the transform matrix and returns just the rotation part of it (I've tried normalizing the result; it does nothing). The matrix
m is a complete transform matrix and contains a translation (and possibly a scale) as well as a rotation. I'm using D3DXMatrixDecompose to do the actual decomposition.