In trying to solve this problem, I tracked down the problem to the conversion of the rotation matrix to quaternion. In particular, consider the following matrix:
-0.02099178 0.9997436 -0.008475631 0 0.995325 0.02009799 -0.09446743 0 0.09427284 0.01041905 0.9954919 0 0 0 0 1
SlimDX.Quaternion.RotationMatrix (which calls D3DXQuaternionRotationMatrix gives a different answer than
SlimDX.Matrix.Decompose (which uses D3DXMatrixDecompose). The answers they give (after being normalized) are:
X Y Z W Quaternion.RotationMatrix -0.05244324 0.05137424 0.002209336 0.9972991 Matrix.Decompose 0.6989997 0.7135442 -0.03674842 -0.03006023
Which are totally different (note the signs of X, Z, and W are different). Note that these aren't q/-q (two quaternions that represent the same rotation); they face completely different directions.
I've noticed that with matrices for rotations very close to that one (successive frames in the animation) that the
Matrix.Decompose version gives a solution that flips around wildly and occasionally goes into the desired position, while the
Quaternion.RotationMatrix version gives solutions that are stable but go in the wrong direction. This is only for the right arm in my animation -- for the left arm, both functions give the correct solution, which is the same quaternion within error tolerances.
This makes me think that there's some sort of numeric instability or weird stuff with signs going on. I tried implementing this and then this, but both gave me a completely incorrect solution (even for the matricies where the SlimDX ones were working correctly) -- maybe the rows and columns are flipped?