0
\$\begingroup\$

I was able to get the Unity ARkit face tracking demo working with an iPhone X to animate a face model in realtime:

Unity Editor screenshot with sloth face model

I then tried to get it to work with my own 3D model, and imported the FBX file with the same blendshapes (though each blendshape has a different name from the original sloth model). By using the hierarchy, ARFaceAnchorManager, and changing the Anchor Prefab I was able to select the model to use:

Inspecting blend shapes of original sloth face

Inspecting blend shapes of my replacement head model

The problem is:

  • the rendered model is flipped along the z axis (it faces away from the camera, as shown below). I've tried to apply a 180 degree rotation to the model in Unity's transform window, but it doesn't seem to do anything.

  • I'm unsure how to map the blendshapes coming from the iPhone app to the blendshapes in the FBX model - where is that done? I couldn't find code that references these blendshapes.

Head facing away from camera instead of towards

\$\endgroup\$
1
\$\begingroup\$

It looks like the code that manipulates the blend shapes is here, in the aptly-named BlendshapeDriver class:

SkinnedMeshRenderer skinnedMeshRenderer;
Dictionary<string, float> currentBlendShapes;

// ...

void FaceUpdated (ARFaceAnchor anchorData)
{
    currentBlendShapes = anchorData.blendShapes;
}

void Update () {

    if (currentBlendShapes != null) {
        foreach(KeyValuePair<string, float> kvp in currentBlendShapes)
        {
            int blendShapeIndex = skinnedMeshRenderer.sharedMesh.GetBlendShapeIndex ("blendShape2." + kvp.Key);
            if (blendShapeIndex >= 0 ) {
                skinnedMeshRenderer.SetBlendShapeWeight (blendShapeIndex, kvp.Value * 100.0f);
            }
        }
    }
}

You can find the list of blend shape names that will come in as currentBlendShapes' kvp.Key field here in Apple's documentation.

With this in hand, we can prepare our own remapping table like so...

[System.Serializable]
public struct Rename {
    [HideInInspector]
    public string originalName;
    public string mappedName;
    public Rename(string originalName) {
        this.originalName = originalName;
        this.mappedName = "";
    }
}

public Rename[] nameRemapping = new Rename[] {
    new Rename("eyeBlinkLeft"),
    new Rename("eyeLookDownLeft"),
    new Rename("eyeLookInLeft"),
    new Rename("eyeLookOutLeft"),
    new Rename("eyeLookUpLeft")
    // ... (Completing this table is left as an exercise for the reader)
};

This gives you a list of strings in the inspector that you can use to specify what the corresponding blendshape names are in your model, like this:

Example of inspector for renaming blend shapes

To save some searching, we can boil this down to a fast index lookup at runtime:

Dictionary<string, int> indexLookup;

void Start() {

    // Contents of existing Start method goes here.

    indexLookup = new Dictionary<string, int>(nameRemapping.Length);

    foreach(var map in nameRemapping) {
        int index = skinnedMeshRenderer.sharedMesh.GetBlendShapeIndex(remapping.mappedName);
        indexLookup.Add(remapping.originalName, index);
    }
}

Now we can replace this line inside Update:

int blendShapeIndex = skinnedMeshRenderer.sharedMesh.GetBlendShapeIndex ("blendShape2." + kvp.Key);

With this line:

int blendShapeIndex = indexLookup[kvp.Key];

As a bonus, by avoiding string concatenation inside our update loop, we've slightly reduced the garbage allocation and improved the efficiency. :)


As for why the head appears facing away, and why rotating it had no effect, I'm not sure what to tell you. The code looks like it's only rotating/repositioning the camera. Two thoughts:

  1. Is it possible you applied your y = 180 rotation to the prefab in your assets folder, instead of the instance in your scene? Prefab orientation is often ignored/overwritten when spawning a scene instance. The symptom of this would be that the 180 you set changes to 0 when you select the scene instance.

  2. Is it possible your SkinnedMeshRenderer has an animation playing a key that's telling it to face forward, overriding the rotation you set? The symptom of this would be that even if you rotate the object at runtime, it resists and resets itself.

    If so, you can either remove the animation, or put the head inside an empty parent object then rotate the parent.

Or the most robust solution might be to open the head in your 3D modelling software and spin it around there, then re-export.

\$\endgroup\$
  • \$\begingroup\$ thanks @DMGregory, I rotated the model in Blender and re-exported as an FBX. But it seems like it doesn't actually export it with a rotated mesh (just an instruction to rotate it), as when Unity imports it it sets the rotation to -180 for the Y axis... which looks fine in the Scene but gets overriden when I click Play and the model is still facing away from the camera. \$\endgroup\$ – skunkwerk Aug 18 at 14:57
  • \$\begingroup\$ You remembered to Apply your transformation to the underlying model using CTRL-A, right? \$\endgroup\$ – DMGregory Aug 18 at 14:58

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.