Euler Angle X is wrong after using Quaternion.LookRotation

Question

I have an direction that I want the local player hub & camera to rotate at. The game handles this by using euler angles which Y represents the hub rotation on the horizontal axis & X represents the camera rotation on the vertical axis.

The min-max range for vertical is -88 to 88 & min-max range for horizontal is 0 to 360

Here is the code I am using the create euler angles that will look onto the direction

Vector2 CalcEuler(Vector3 dst, Vector3 src)
{
    //dst is target position, src is maincamera position
    Quaternion quat = Quaternion.LookRotation((dst - 
                                            src), Vector3.right);
    Vector3 euler = quat.get_eulerAngles();
    return { euler.y, euler.x };
}

the horizontal axis(euler.y) returned from CalcEuler is perfect and rotates the hub to lock onto the specified direction. However the vertical axis(euler.x) returned from CalcEuler only works if the direction.Y value is the same as src.Y. If it isnt then itll shoot up into really high numbers past 200 and even sometimes at 359.99.

What could I be doing wrong?

EDIT: I found something some guy did on github for his rotation. After doing Quaternion.Lookrotation he did this

euler.x = (src.y < dst.y) ? (-360.f + euler.x) : euler.x;

however this does not change anything for me

SOLVED: Both answers fix the problem

Answer 1

If you want to enforce angular limits, keep it in terms of angles from the start. Don't count on an automatic conversion from quaternions to Euler angles to give you angles in a specific range, because there are many Euler angle triplets corresponding to any one quaternion, and any mapping we pick will be wrong for at least one use case.

public static Vector2 GetClampedEuler(Vector3 lookTarget, Vector3 lookOrigin, float maxPitch) {
    Vector3 direction = (lookTarget - lookOrigin).normalized;

    // Get rotation in horizontal plane in range -180...180
    float yaw = Mathf.Atan2(direction.x, direction.z) * Mathf.Rad2Deg;
    // Remap to 0...360
    if (yaw < 0f) yaw += 360f;

    // Get angle below horizontal in range -90...90
    float pitch = Mathf.Asin(-direction.y) * Mathf.Rad2Deg;
    
    // Enforce pitch limits
    return new Vector2(yaw, Mathf.Clamp(pitch, -maxPitch, maxPitch));
}

Answer 2

If you want to isolate the heading of a look direction then zero out the up and use the atan2 function directly.

vector3 dir = dst - src;
heading = atan2(dir.y, dir.x); //assuming the up/down is along z and 0 heading is along x, swap axis as appropriate
pitch = atan2(hypot(dir.x, dir.y), dir.z);

Though you will need to take care when hypot(dir.x, dir.y) is very small, aka when the pitch angle would exceed the bounds.

Though frankly I dislike computing back to angles using the arctan/arccos just to be put into a rotate which will put it straight into a cos and sin.

instead you can just pass the look direction directly through to whatever does the rotating hand have it figure out the correct transform operation based on just (dst - src).normalized()