How does Unity's FromToRotation work? A major part of its behavior seems undocumented

Question

All Unity's documentation states about what FromToRotation does is:

Creates a rotation which rotates from fromDirection to toDirection.

This seems to leave a major part undefined - what about the other axes?

I can't help but think of the saying - If it isn't documented, the behavior is undefined.

And there seems to be no documentation of what happens to the rest of the axes. Theoretically you could spin them around toDirection as much as you'd like and still get what you were promised.

So, what exactly happens to the other axes when you use FromToRotation?

Answer 1

FromToRotation chooses the shortest possible rotation that maps fromDirection to toDirection.

That is, the rotation about the axis

axis = Vector3.Cross(fromDirection, toDirection);

through an angle of

angleRadians = Mathf.Acos(Vector3.Dot(
                  fromDirection.normalized, 
                  toDirection.normalized
               ));

This does become indeterminate if the two directions are exactly opposite: then a 180 degree rotation about any axis perpendicular to them is an equally short option, so the engine has to choose one arbitrarily.

I'm currently on mobile, so I have not tested which axis it uses in this special case, but I'd expect it uses a routine something similar to:

var right = Vector3.Cross(Vector3.up, fromDirection);
if (Math.Approximately(right.sqrMagnitude, 0)) {
    axis = Vector3.forward;
} else {
    axis = Vector3.Cross(fromDirection, right);
}

This gives the y+ axis if the two inputs are in the xz plane, the axis closest to y+ if the two inputs aren't themselves vertical, and the z+ axis if the two inputs are vertical. That choice is arbitrary, but seems to make sense for typical 3D navigation use cases.