How to programatically retarget animations from one skeleton to another?

Question

I'm trying to write code to transfer animations that were designed for one skeleton to look correct on another skeleton. The source animations consist only of rotations except for translations on the root (they're the mocap animations from the CMU motion capture database). Many 3D applications (eg Maya) have this facility built-in, but I'm trying to write a (very simple) version of it for my game.

I've done some work on bone mapping, and because the skeletons are hierarchically similar (bipeds), I can do 1:1 bone mapping for everything but the spine (can work on that later). The problem, however, is that the base skeleton/bind poses are different, and the bones are different scales (shorter/longer), so if I just copy the rotation straight over it looks very strange.

I've attempted a number of things similar to lorancou's solution below to no avail (ie multiplying each frame in the animation by a bone-specific multiplier). If anyone has any resources on stuff like this (papers, source code, etc), that would be really helpful.

Answer 1

The problem was one of numerical stability. Approx 30 hours of work on this over the course of 2 months, only to figure out I was doing it right from the very start. When I ortho-normalized the rotation matrices before plugging them into the retarget code, the simple solution of multiplying source * inverse(target) worked perfectly. Of course, there's a lot more to retargeting than that (in particular, taking into account the different shapes of the skeleton, ie shoulder width, etc). Here's the code I'm using for the simple, naieve approach, if anyone is curious:

    public static SkeletalAnimation retarget(SkeletalAnimation animation, Skeleton target, string boneMapFilePath)
    {
        if(animation == null) throw new ArgumentNullException("animation");
        if(target == null) throw new ArgumentNullException("target");

        Skeleton source = animation.skeleton;
        if(source == target) return animation;

        int nSourceBones = source.count;
        int nTargetBones = target.count;
        int nFrames = animation.nFrames; 
        AnimationData[] sourceData = animation.data;
        Matrix[] sourceTransforms = new Matrix[nSourceBones];
        Matrix[] targetTransforms = new Matrix[nTargetBones];
        AnimationData[] temp = new AnimationData[nSourceBones];
        AnimationData[] targetData = new AnimationData[nTargetBones * nFrames];

        // Get a map where map[iTargetBone] = iSourceBone or -1 if no such bone
        int[] map = parseBoneMap(source, target, boneMapFilePath);

        for(int iFrame = 0; iFrame < nFrames; iFrame++)
        {
            int sourceBase = iFrame * nSourceBones;
            int targetBase = iFrame * nTargetBones;

            // Copy the root translation and rotation directly over
            AnimationData rootData = targetData[targetBase] = sourceData[sourceBase];

            // Get the source pose for this frame
            Array.Copy(sourceData, sourceBase, temp, 0, nSourceBones);
            source.getAbsoluteTransforms(temp, sourceTransforms);

            // Rotate target bones to face that direction
            Matrix m;
            AnimationData.toMatrix(ref rootData, out m);
            Matrix.Multiply(ref m, ref target.relatives[0], out targetTransforms[0]);
            for(int iTargetBone = 1; iTargetBone < nTargetBones; iTargetBone++)
            {
                int targetIndex = targetBase + iTargetBone;
                int iTargetParent = target.hierarchy[iTargetBone];
                int iSourceBone = map[iTargetBone];
                if(iSourceBone <= 0)
                {
                    targetData[targetIndex].rotation = Quaternion.Identity;
                    Matrix.Multiply(ref target.relatives[iTargetBone], ref targetTransforms[iTargetParent], out targetTransforms[iTargetBone]);
                }
                else
                {
                    Matrix currentTransform, inverseCurrent, sourceTransform, final, m2;
                    Quaternion rot;

                    // Get the "current" transformation (transform that would be applied if rot is Quaternion.Identity)
                    Matrix.Multiply(ref target.relatives[iTargetBone], ref targetTransforms[iTargetParent], out currentTransform);
                    Math2.orthoNormalize(ref currentTransform);
                    Matrix.Invert(ref currentTransform, out inverseCurrent);
                    Math2.orthoNormalize(ref inverseCurrent);

                    // Get the final rotation
                    Math2.orthoNormalize(ref sourceTransforms[iSourceBone], out sourceTransform);
                    Matrix.Multiply(ref sourceTransform, ref inverseCurrent, out final);
                    Math2.orthoNormalize(ref final);
                    Quaternion.RotationMatrix(ref final, out rot);

                    // Calculate this bone's absolute position to use as next bone's parent
                    targetData[targetIndex].rotation = rot;
                    Matrix.RotationQuaternion(ref rot, out m);
                    Matrix.Multiply(ref m, ref target.relatives[iTargetBone], out m2);
                    Matrix.Multiply(ref m2, ref targetTransforms[iTargetParent], out targetTransforms[iTargetBone]);
                }
            }
        }

        return new SkeletalAnimation(target, targetData, animation.fps, nFrames);
    }

Answer 2

I believe that your easiest option is simply to match the original bind pose with your new skeleton if you've got the possibility (if your new skeleton isn't skinned yet).

If you can't do that, here's something you could try. This is just intuition, I'm probably overlooking lots of things, but it might help you to find the light. For each bone:

• In your "old" bind pose, you've got one quaternion that describes the relative rotation of this bone compared to its parent bone. Here's a hint for how to find it. Let's call it q_old.

• Ibid. for your "new" bind pose, let's call it q_new.

• You can find the relative rotation from the "new" bind pose to the "old" bin pose, as described here. That's q_new_to_old = inverse(q_new) * q_old.

• Then in one animation key, you've got your one quaternion that transforms that bone from the "old" bind pose to an animated pose. Let's call this one q_anim.

Instead of using q_anim directly, try using q_new_to_old * q_anim. This should "cancel" the orientation differences between the bind poses, before applying the animation.

It might do the trick.

EDIT

Your code above seems to follow the logic I'm describing here, but something's inverted. Instead of doing this:

multipliers[iSourceBone] = Quaternion.Invert(sourceBoneRot) * targetBoneRot;

You could try that:

multipliers[iSourceBone] = Quaternion.Invert(targetBoneRot) * sourceBoneRot;

I think that you need to transform from your target to your source before applying the source animation, to get the same final orientation.