How to decompose a 4x4 TRS transformation matrix with negative scale?

Question

As the title says I need to decompose 4x4 TRS transformation matrices and extract the proper scale vectors and the proper rotation vectors (or rotation quaternions).

I know how to extract those information when the upper 3x3 matrix determinant is not negative. The problem is that those matrices can have that negative determinant and as far as I could understand, this negative determinant indicates a flip or mirrored transformation.

What do I have to do to extract the proper values for scale and rotation in those cases?

Answer 1

Reopened this to give you a Unity implementation of the method described in the Q&A I linked above:

public static class MatrixHelpers
{
    public static Vector3 GetTranslation(this Matrix4x4 matrix) { 
        return matrix.GetColumn(3); 
    }

    public static bool TryGetRotation(this Matrix4x4 matrix, out Quaternion rotation) {
        Vector3 forward = matrix.GetColumn(2);
        if (forward.sqrMagnitude == 0f) {
            rotation = Quaternion.identity;
            return false;
        }

        Vector3 up = matrix.GetColumn(1);
        if (up.sqrMagnitude == 0f) {
            rotation = Quaternion.LookRotation(forward);
            return false;
        }

        rotation = Quaternion.LookRotation(forward, up);
        return true;
    }

    public static bool TryDecomposeTRS(this Matrix4x4 matrix, 
           out Vector3 translation, out Quaternion rotation, out Vector3 scale) {
        translation = matrix.GetTranslation();
        scale = matrix.lossyScale;

        return (matrix.TryGetRotation(out rotation) && matrix.ValidTRS());
    }
}

With this you can write:

if (!someMatrix.TryDecomposeTRS(out Vector3 t, out Quaternion r, out Vector3 s)) {
    Debug.LogWarning("Input matrix was not in TRS form."
                     + " It may include shear or projection."
                     + " Results will be approximate.");
}
// TODO: Do something useful with the decomposed translation, rotation, and scale.

You can probably see why there's not much in the way of external libraries for this - you can do everything you need in just a few lines with the built-in Unity types.