LookRotation function for Z-up, right handed system

Question

I am rendering graphics myself using OpenGL, therefore I need to implement math library myself. I am looking for function Quaternion::LookRotation(forward, up), given a forward and up vector, it shall create a quaternion rotating in this direction.

I've found this answer http://answers.unity3d.com/questions/467614/what-is-the-source-code-of-quaternionlookrotation.html for LookRotation and it works well providing we are using Unity's coordinate system (left handed): Forward: (0, 0, 1), Up: (0, 1, 0), Right (1, 0, 0).

In this coordinate system the function works as expected:

var dir = new Vector3(1, 0, 1).normalized;
var quaternion = Quaternion.LookRotation(dir, Vector3.Up); // Up == (0, 1, 0)
var backToDirection = quaternion * Vector3.Forward; // Forward == (0, 0, 1)
// backToDirection equals to dir

However, in my "engine", I am using right handed, Z-up coordinate system: Forward: (1, 0, 0), Up: (0, 0, 1), Right: (0, -1, 0).

Unless I am wrong, the very same code (with corrected LookRotation) should still work:

var dir = new Vector3(1, 0, 1).normalized;
var quaternion = Quaternion.LookRotation(dir, Vector3.Up); // new Up == (0, 0, 1)
var backToDirection = quaternion * Vector3.Forward; // new Forward == (1, 0, 0)
// backToDirection should be equal to dir, but obviously `LookRotation` needs to be changed

However, for this to work, LookRotation needs to be adjusted accordingly. The source code for LookRotation posted at answers.unity3d doesn't explain why, neither I could find a good explanation how to do it. Could anyone give me a hint what needs to be changed there in order to make it work in a different coordinate system?

Answer 1

First, we can transform your coordinates into Unity's coordinates:

Quaternion LookRotation(Vector3 myForward, Vector3 myUp) {

    var uForward = new Vector3(-myForward.y, myForward.z, myForward.x);
    var uUp      = new Vector3(-     myUp.y,      myUp.z,      myUp.x);

...then we can invoke the Unity code, unchanged:

    var uQuaternion = UnityLookRotation(uForward, uUp);

...then transform the resulting quaternion back into your coordinate system. That means remapping the axes to match yours, and negating the angle since we're now rotating in a right-handed instead of left-handed scheme.

Since the xyz components are just a unit vector along the axis of rotation times the sine of half the angle, and the w component is the cosine of half the angle, that gives us:

    var myQuaternion = new Quaternion(
        -uQuaternion.z, // Our +x is Unity's +z, and we negate the angle
         uQuaternion.x, // Our +y is Unity's -x, and we negate the angle
        -uQuaternion.y, // Our +z is Unity's +y, and we negate the angle
         uQuaternion.w  // cosine(angle) = cosine(-angle), so w remains
    );

    return myQuaternion;
}

This should work out of the box, and we can leave it for the compiler to inline and optimize it for us. Or, you can go through the steps of propagating the subscript and sign changes through the Unity method so that it bakes-in the effect of these transformations for you.