# How to programatically retarget animations from one skeleton to another?

I'm trying to write code to transfer animations that were designed for one skeleton to look correct on another skeleton. The source animations consist only of rotations except for translations on the root (they're the mocap animations from the CMU motion capture database). Many 3D applications (eg Maya) have this facility built-in, but I'm trying to write a (very simple) version of it for my game.

I've done some work on bone mapping, and because the skeletons are hierarchically similar (bipeds), I can do 1:1 bone mapping for everything but the spine (can work on that later). The problem, however, is that the base skeleton/bind poses are different, and the bones are different scales (shorter/longer), so if I just copy the rotation straight over it looks very strange.

I've attempted a number of things similar to lorancou's solution below to no avail (ie multiplying each frame in the animation by a bone-specific multiplier). If anyone has any resources on stuff like this (papers, source code, etc), that would be really helpful.

• How do you expect me to ignore the tail and the thing between the legs? :P – kaoD Apr 9 '12 at 13:47
• @kaoD If you have to ask, the skeleton is rooted at (0,0) so there's a fake bone there. As for the tail... everyone knows life is better if you have a tail. I've always thought it would be efficient for things like carrying coffee cups and balancing on tree limbs. – Robert Fraser Apr 9 '12 at 23:04
• I have seen a real time demo of this where a kinect was used to animate a model displayed in xna. Think the code was on an open source site. Will search... – George Duckett Apr 11 '12 at 19:29
• forums.create.msdn.com/forums/p/102742/609352.aspx – George Duckett Apr 11 '12 at 19:39
• I suspect that your problem is more with the distinct bind poses than with the bones scaling, you could try isolating that. For instance start from the original skeleton, scale a few bones on it to create a new skeleton, and see if your algorithm breaks with this one. If it doesn't, then restart from the original skeleton but this time don't scale bones, just rotate them and see if your algorithm breaks. If it does, then yeah there's probably an additional transform to perform somewhere. – Laurent Couvidou Apr 14 '12 at 12:51

The problem was one of numerical stability. Approx 30 hours of work on this over the course of 2 months, only to figure out I was doing it right from the very start. When I ortho-normalized the rotation matrices before plugging them into the retarget code, the simple solution of multiplying source * inverse(target) worked perfectly. Of course, there's a lot more to retargeting than that (in particular, taking into account the different shapes of the skeleton, ie shoulder width, etc). Here's the code I'm using for the simple, naieve approach, if anyone is curious:

    public static SkeletalAnimation retarget(SkeletalAnimation animation, Skeleton target, string boneMapFilePath)
{
if(animation == null) throw new ArgumentNullException("animation");
if(target == null) throw new ArgumentNullException("target");

Skeleton source = animation.skeleton;
if(source == target) return animation;

int nSourceBones = source.count;
int nTargetBones = target.count;
int nFrames = animation.nFrames;
AnimationData[] sourceData = animation.data;
Matrix[] sourceTransforms = new Matrix[nSourceBones];
Matrix[] targetTransforms = new Matrix[nTargetBones];
AnimationData[] temp = new AnimationData[nSourceBones];
AnimationData[] targetData = new AnimationData[nTargetBones * nFrames];

// Get a map where map[iTargetBone] = iSourceBone or -1 if no such bone
int[] map = parseBoneMap(source, target, boneMapFilePath);

for(int iFrame = 0; iFrame < nFrames; iFrame++)
{
int sourceBase = iFrame * nSourceBones;
int targetBase = iFrame * nTargetBones;

// Copy the root translation and rotation directly over
AnimationData rootData = targetData[targetBase] = sourceData[sourceBase];

// Get the source pose for this frame
Array.Copy(sourceData, sourceBase, temp, 0, nSourceBones);
source.getAbsoluteTransforms(temp, sourceTransforms);

// Rotate target bones to face that direction
Matrix m;
AnimationData.toMatrix(ref rootData, out m);
Matrix.Multiply(ref m, ref target.relatives[0], out targetTransforms[0]);
for(int iTargetBone = 1; iTargetBone < nTargetBones; iTargetBone++)
{
int targetIndex = targetBase + iTargetBone;
int iTargetParent = target.hierarchy[iTargetBone];
int iSourceBone = map[iTargetBone];
if(iSourceBone <= 0)
{
targetData[targetIndex].rotation = Quaternion.Identity;
Matrix.Multiply(ref target.relatives[iTargetBone], ref targetTransforms[iTargetParent], out targetTransforms[iTargetBone]);
}
else
{
Matrix currentTransform, inverseCurrent, sourceTransform, final, m2;
Quaternion rot;

// Get the "current" transformation (transform that would be applied if rot is Quaternion.Identity)
Matrix.Multiply(ref target.relatives[iTargetBone], ref targetTransforms[iTargetParent], out currentTransform);
Math2.orthoNormalize(ref currentTransform);
Matrix.Invert(ref currentTransform, out inverseCurrent);
Math2.orthoNormalize(ref inverseCurrent);

// Get the final rotation
Math2.orthoNormalize(ref sourceTransforms[iSourceBone], out sourceTransform);
Matrix.Multiply(ref sourceTransform, ref inverseCurrent, out final);
Math2.orthoNormalize(ref final);
Quaternion.RotationMatrix(ref final, out rot);

// Calculate this bone's absolute position to use as next bone's parent
targetData[targetIndex].rotation = rot;
Matrix.RotationQuaternion(ref rot, out m);
Matrix.Multiply(ref m, ref target.relatives[iTargetBone], out m2);
Matrix.Multiply(ref m2, ref targetTransforms[iTargetParent], out targetTransforms[iTargetBone]);
}
}
}

return new SkeletalAnimation(target, targetData, animation.fps, nFrames);
}

• is the code on this page been updated since its been written? Having a hard time trying to understand without the context of the engine that uses it. I'm trying to do animation retargeting as well. Would be great to have some pseudo code of the steps of how to handle retargeting. – SketchpunkLabs Aug 21 '19 at 6:35

I believe that your easiest option is simply to match the original bind pose with your new skeleton if you've got the possibility (if your new skeleton isn't skinned yet).

If you can't do that, here's something you could try. This is just intuition, I'm probably overlooking lots of things, but it might help you to find the light. For each bone:

• In your "old" bind pose, you've got one quaternion that describes the relative rotation of this bone compared to its parent bone. Here's a hint for how to find it. Let's call it q_old.

• Ibid. for your "new" bind pose, let's call it q_new.

• You can find the relative rotation from the "new" bind pose to the "old" bin pose, as described here. That's q_new_to_old = inverse(q_new) * q_old.

• Then in one animation key, you've got your one quaternion that transforms that bone from the "old" bind pose to an animated pose. Let's call this one q_anim.

Instead of using q_anim directly, try using q_new_to_old * q_anim. This should "cancel" the orientation differences between the bind poses, before applying the animation.

It might do the trick.

EDIT

Your code above seems to follow the logic I'm describing here, but something's inverted. Instead of doing this:

multipliers[iSourceBone] = Quaternion.Invert(sourceBoneRot) * targetBoneRot;


You could try that:

multipliers[iSourceBone] = Quaternion.Invert(targetBoneRot) * sourceBoneRot;


I think that you need to transform from your target to your source before applying the source animation, to get the same final orientation.

• The bind poses of both the sources and the targets are going to vary, which is why I'm implementing this :-) . Indeed, multiplying by the inverse of the target rotation was the first thing I tried. I tried recalculating the bone rotations, as per your suggestion, but the result was the same. Here's a video of what's going wrong: youtube.com/watch?v=H6Qq37TM4Pg – Robert Fraser Apr 16 '12 at 5:09
• Are you sure you're always expressing your rotations relatively to a parent bone? Seeing your video, it looks like you're using an absolute/world rotation somewhere, where you should use a relative-to-parent rotation instead. – Laurent Couvidou Apr 16 '12 at 15:57
• Yes, I'm fairly sure I'm using the relative transformations here (I've tried with absolutes, it looks much stranger).I updated the OP with the code I was using for this video. Rather than trying to debug it this way, I'd rather see some source code or tutorials where it was done successfully, then I can figure out what I'm doing wrong. – Robert Fraser Apr 16 '12 at 16:31
• Sure, but maybe there's no tutorial to do exactly that :) I think you inverted something in your code above, I'll edit my answer. – Laurent Couvidou Apr 16 '12 at 21:29
• I've tried many ways, and none of the worked. I'm going to try actually calculating the global rotations on a per-frame basis to see what's going wrong. Thanks for your help, though; I'll give you the 100 rep. – Robert Fraser Apr 18 '12 at 23:37