Changing the 3D model of the Unity ARkit face tracking demo

Question

I was able to get the Unity ARkit face tracking demo working with an iPhone X to animate a face model in realtime:

I then tried to get it to work with my own 3D model, and imported the FBX file with the same blendshapes (though each blendshape has a different name from the original sloth model). By using the hierarchy, ARFaceAnchorManager, and changing the Anchor Prefab I was able to select the model to use:

The problem is:

• the rendered model is flipped along the z axis (it faces away from the camera, as shown below). I've tried to apply a 180 degree rotation to the model in Unity's transform window, but it doesn't seem to do anything.

• I'm unsure how to map the blendshapes coming from the iPhone app to the blendshapes in the FBX model - where is that done? I couldn't find code that references these blendshapes.

Answer 1

It looks like the code that manipulates the blend shapes is here, in the aptly-named BlendshapeDriver class:

SkinnedMeshRenderer skinnedMeshRenderer;
Dictionary<string, float> currentBlendShapes;

// ...

void FaceUpdated (ARFaceAnchor anchorData)
{
    currentBlendShapes = anchorData.blendShapes;
}

void Update () {

    if (currentBlendShapes != null) {
        foreach(KeyValuePair<string, float> kvp in currentBlendShapes)
        {
            int blendShapeIndex = skinnedMeshRenderer.sharedMesh.GetBlendShapeIndex ("blendShape2." + kvp.Key);
            if (blendShapeIndex >= 0 ) {
                skinnedMeshRenderer.SetBlendShapeWeight (blendShapeIndex, kvp.Value * 100.0f);
            }
        }
    }
}

You can find the list of blend shape names that will come in as currentBlendShapes' kvp.Key field here in Apple's documentation.

With this in hand, we can prepare our own remapping table like so...

[System.Serializable]
public struct Rename {
    [HideInInspector]
    public string originalName;
    public string mappedName;
    public Rename(string originalName) {
        this.originalName = originalName;
        this.mappedName = "";
    }
}

public Rename[] nameRemapping = new Rename[] {
    new Rename("eyeBlinkLeft"),
    new Rename("eyeLookDownLeft"),
    new Rename("eyeLookInLeft"),
    new Rename("eyeLookOutLeft"),
    new Rename("eyeLookUpLeft")
    // ... (Completing this table is left as an exercise for the reader)
};

This gives you a list of strings in the inspector that you can use to specify what the corresponding blendshape names are in your model, like this:

To save some searching, we can boil this down to a fast index lookup at runtime:

Dictionary<string, int> indexLookup;

void Start() {

    // Contents of existing Start method goes here.

    indexLookup = new Dictionary<string, int>(nameRemapping.Length);

    foreach(var map in nameRemapping) {
        int index = skinnedMeshRenderer.sharedMesh.GetBlendShapeIndex(remapping.mappedName);
        indexLookup.Add(remapping.originalName, index);
    }
}

Now we can replace this line inside Update:

int blendShapeIndex = skinnedMeshRenderer.sharedMesh.GetBlendShapeIndex ("blendShape2." + kvp.Key);

With this line:

int blendShapeIndex = indexLookup[kvp.Key];

As a bonus, by avoiding string concatenation inside our update loop, we've slightly reduced the garbage allocation and improved the efficiency. :)

---

As for why the head appears facing away, and why rotating it had no effect, I'm not sure what to tell you. The code looks like it's only rotating/repositioning the camera. Two thoughts:

1. Is it possible you applied your y = 180 rotation to the prefab in your assets folder, instead of the instance in your scene? Prefab orientation is often ignored/overwritten when spawning a scene instance. The symptom of this would be that the 180 you set changes to 0 when you select the scene instance.

2. Is it possible your SkinnedMeshRenderer has an animation playing a key that's telling it to face forward, overriding the rotation you set? The symptom of this would be that even if you rotate the object at runtime, it resists and resets itself.

   If so, you can either remove the animation, or put the head inside an empty parent object then rotate the parent.

Or the most robust solution might be to open the head in your 3D modelling software and spin it around there, then re-export.