Rotating only 2 axes of rotation quaternion to "point" in a direction

Question

Note: I am using the Unity DOTS ECS framework, which contains less maths functions, but I can probably implement the same functions from standard unity if the code is public. This question is not specific to unity though, I think.

I'm trying to have a player be able to run around the inside of a sphere such that their local up direction is always pointing at the centre of the sphere. So, when they run "forward" they naturally run in an orbit around the sphere surface.

I already have an artificial gravity system implemented, that applies a force pushing the player towards the sphere walls.

I calculate the new up direction, and then calculate the new forward direction from that. Feeding these into the built-in quaternion.LookRotation(newForward, newUp) function does orient my player towards the centre, but this also updates the local y-axis rotation of the player, meaning I can't turn the player, and it spins around when you cross each hemisphere.

float3 centerDir = math.normalize(sphereCenter.Value - translation.Value);

quaternion newRotation = rotation.Value;

float3 newUp = centerDir;
float3 newForward = mathx.Forward(newUp);

newRotation = quaternion.LookRotation(newForward, newUp);

rotation.Value = newRotation;

I then tried to negate the Y axis rotation by trying to find the difference in y axis rotation, and multiplying the rotation by another quaternion rotated on Y only by the negative of this angle. But I'm pretty sure I have the wrong vectors in my calculation to find this angle.

This had some success, but the player starts 90 degrees rotated and did strange things as I moved near to the axes. Where forward would bend towards the axis and start spinning.

float3 centerDir = math.normalize(sphereCenter.Value - translation.Value);

quaternion newRotation = rotation.Value;

float3 newUp = centerDir;
float3 newForward = mathx.Forward(newUp);

newRotation = quaternion.LookRotation(newForward, newUp);

float angleDiff = mathx.Angle(ltw.Forward, newForward);
quaternion negateYRotation = quaternion.RotateY(-angleDiff);

rotation.Value = math.mul(newRotation, negateYRotation);

I've been banging my head against a wall for days with this. I even tried converting all of the quaternions into Euler angles and setting the y angle to the original y angle, but this also affects the other angles so the player is no longer oriented correctly. (I assume this might be to do with the z axis being applied after the y axis).

Any help with this would be much appreciated.

Answer 1

Following DMGregory's excellent answer and posts linked, I now understand the issue. I have implemented his function provided in one of those answers, and it works beautifully.

This is my solution:

I have added a new static function to my extension class, for easy access.

public static class mathx
{
    public static quaternion LookRotationExactUp(float3 approximateForward, float3 exactUp)
    {
        quaternion rotateZToUp = quaternion.LookRotation(exactUp, -approximateForward);
        quaternion rotateYToZ = quaternion.RotateX(math.radians(90));

        return math.mul(rotateZToUp, rotateYToZ);
    }
}

And the working implementation:

[UpdateAfter(typeof(TransformSystemGroup))]
public class RotateUpToSphereCenterSystem : SystemBase
{
    protected override void OnUpdate()
    {
        float3 sphereCenter = new float3(0, 0, 0);
        float deltaTime = Time.DeltaTime;
        float rotationSpeed = 5;

        Entities
            .ForEach((ref Rotation rotation, ref MovementData movement, in LocalToWorld ltw) =>
            {
                // Only rotate if not at exact center of sphere
                if (!ltw.Position.Equals(sphereCenter))
                {
                    float3 centerDir = math.normalize(sphereCenter - ltw.Position);

                    float3 newUp = centerDir;

                    //Guarantee exact up direction and maintain forward as close as possible
                    movement.targetRotation = mathx.LookRotationExactUp(ltw.Forward, newUp);

                    //Smoothly rotate to the targetRotation
                    rotation.Value = math.slerp(rotation.Value, movement.targetRotation, deltaTime * rotationSpeed);
                }
            }).ScheduleParallel();        
    }
}
```

Answer 2

I get a lot of mileage out of this trick (see several past answers):

// Find what direction points opposite our current facing direction (local z-).
float3 backward = math.mul(rotation.Value, new float3(0, 0, -1));
// Find the new up vector we want to rotate our local y+ axis to match.
float3 newUp = math.normalize(sphereCenter.Value - translation.Value);

// Compute a rotation that points our local z+ vector exactly at newUp,
// and our local y+ vector as close as possible to backward.
quaternion zToUp = quaternion.LookRotation(newUp, backward);

// Pitch forward 90 degrees, so now y+ points toward newUp,
// and z+ points perpendicular to this, as close as possible to our old forward.
quaternion yToZ =  quaternion.Euler(math.PI/2, 0, 0));
quaternion newRotation = math.mul(zToUp, yToZ);