# Motion Retargeting from a BVH File to a Rigged Mesh in Unity with Arbitrary Rest Pose

I want to perform motion retargeting from a .bvh file (from lafan2 dataset) to a rigged mesh in Unity (which is John Lemon in this tutorial resource).

I have overcome most of the problems, but this last one got me: I need to somehow calculate the rotation between John Lemon and lafan's rest poses. Now, these two rest poses look different, are in different coordinate systems, and it kills me to identify the orientation of each bones, especially in a .bvh file, since they don't have any meshes and the pose was positioned in a non-human manner, meaning that it was hard to tell what is the "twist" of each joint.

Here are the pictures of these "nice" rest poses:

FYI, these are what their T-poses look like, which are, sadly, not their rest poses:

What I have right now: the .bvh file data along with global orientation, location for lafan, and bone structures consists of gameObjects for John. I guess theoretically I can identify the bone orientation manually, but I am seeking for advices. Are there any easier ways to calculate the rotation between these two rest poses?

What I have tried: I first tried calculating the rotation between vectors. And later I learned that treating the bones as vectors is not cool, because I can only get the "swing" part of the rotation by that, and the "twist" part of the desired rotation was lost. As a result, my character's joints seem to be at the right position, but all his joints are twisted along the bone direction. You can see how Jonh's spine and limbs twist

I kinda fixed it myself. Solution is simple: I just hand-tuned all the required twist angles in Unity. There is a hidden problem though: the order of the 'twist' and 'swing' rotation matters. Result looks much better now: But as you can see, the head orientation is still a little off. I'll fixed it with IK, since it's more of a data problem than a bug.

Edit: Actually, it was a bug. Now its head is way better, even without IK.

Here are the formulars to explain what I was doing. I hate equations, but we will have to figure it out to solve the problem.

What I have is the global orientation, location (calculated) from a .bvh file, and a skeleton structure of gameObjects in Unity. At the end of the day, I want unity mesh (which is John) to pose exactly like what the .bvh file (which is lafan) is doing, so thier final orientation should align: $$O_{JL} = O_{lafan}$$ Considering that these two skeletons have different rest poses in different coordinate systems, we have to transfer the data all into one coordinate system. In this case, we will put them into Unity's system.

After that, we need to face the aforementioned rest pose problem. Assume that we already know how to represent the rest pose of lafan in Unity, we have: $$Q_1 \times O_{JL}^{rest} = Q_2 \times O_{lafan}^{rest}$$ where $$\Q_1\$$ is what I wanted, $$\O_{JL}^{rest}\$$ are all Quaternion.identity, $$\Q_2\$$ is known by FK calculation, but $$\O_{lafan}^{rest}\$$ was hard to calculate. So I made another assumption: assume that we know the rotation between two rest poses already, then I can just apply the ratation to $$\Q_2\$$, and get $$\Q_1\$$. How can I get the roation between two rest poses then?

Well, according to my question, I actually already knew the swing part of the rotation. I just need the twist part. Also, like I mentioned, given the fact that the twist part was probably a rotation just around one axis, it's not hard to tune it manually. If we have the swing and the twist rotation, then we have: $$O_{lafan}^{rest} = Q' \times O_{JL}^{rest} = Q_t \times Q_s \times O_{JL}^{rest}$$

where $$\Q'\$$ is the roation between, and is composed of $$\Q_s\$$ and $$\Q_t\$$, multiplied in that order. Put that back into the equation above, we have: $$Q_1 = Q_2 \times Q_t \times Q_s$$ meaning that with $$\Q_s\$$ and $$\Q_t\$$, we have $$\Q_1\$$. Be careful, the order counts. Other then that, we are good to go.

I may make mistakes here, so feel free to point them out...