Character rotation jitters when looking around

Question

I want my character to walk around sphere like a planet, so in my script I rotate it according to the normal of a raycast and I also rotate my character on the Y axis when I look around with the mouse. When press play, I can look in any direction in front of me, left and right, but when my characters Y rotation gets closer to 180, the X and Z rotation start jittering.

    RaycastHit hit;
    Vector3 rayStart = new Vector3(0,-1,0) + transform.position;
    if (Physics.Raycast(rayStart, -transform.up, out hit, Mathf.Infinity))
    {

        Vector3 rotation = new Vector3(Quaternion.LookRotation(hit.normal).x,0, Quaternion.LookRotation(hit.normal).z);
        transform.Rotate(0, Input.GetAxis("Mouse X") * sensitivityX, 0);
        transform.eulerAngles = new Vector3(rotation.x,transform.eulerAngles.y,rotation.z);
    }

I also tried transform.rotation = Quaternion.Euler(rotation.x,transform.eulerAngles.y,rotation.z); but it changes nothing. If I remove the 3rd line, it works, so i'm guessing i'm assigning the rotation the wrong way ? Both the line to rotate the character according to the camera and the one to rotate it around the planet work individually, but not together.

Answer 1

This line extracts just 2 imaginary components out of a 4-dimensional quaternion:

Vector3 rotation = new Vector3( 
                          Quaternion.LookRotation(hit.normal).x,
                          0, 
                          Quaternion.LookRotation(hit.normal).z
                   );

In general, it is not safe to extract individual components of a quaternion like this. They're not angles like Euler angle triplets. They're individual components of a point on a unit sphere in 4 dimensions, so they don't really make sense without all 4 components.

Each quaternion component ranges from -1 to 1, so it's not really sensible to use as an angle in degrees here:

transform.eulerAngles = new Vector3(rotation.x,transform.eulerAngles.y,rotation.z);

It looks like instead you want to do something like this - using a trick I've covered in several past answers:

// Helper function to make an orientation whose local y+ direction matches "exactUp"
// and whose local z+ direction points as close as possible to "approximateForward". 
Quaternion TurretLookRotation(Vector3 approximateForward, Vector3 exactUp)
{
    Quaternion zToUp = Quaternion.LookRotation(exactUp, -approximateForward);
    Quaternion yToZ = Quaternion.Euler(90, 0, 0);

    return zToUp * yToZ;
}

// ...

// Start your raycast below your position relative to your current orientation.
Vector3 rayStart = Transform.TransformPoint(Vector3.down);
if (Physics.Raycast(rayStart, -transform.up, out RaycastHit hit, Mathf.Infinity))
{
    // Match our vertical orientation to the surface under us, while
    // preserving our forward facing direction as best that allows.
    Quaternion matchSurface = TurretLookRotation(transform.forward, hit.normal);

    // Compute the rotation about our vertical axis according to mouse movement.
    Quaternion twist = Quaternion.Euler(0, Input.GetAxis("MouseX") * sensitivityX, 0);
    
    // Compose these two rotations to get our resulting orientation.
    transform.rotation = matchSurface * twist;
}