Player rotation recalculation causes jittering on edges

Question

When my player walks around a circular planet everything works well, but when my planet is a shape like a cube or a dodecahedron, even with smoothed edges, my character jitters if I walk slowly on the edge.

That is because I raycast downward and set my characters rotation to be perpendicular with the normal. If im on the edge, the ray hits a different side of the shape than the one I stand on, causing my player to rotate to that new side, even if the angle between the face I stand on and the one I hit is 179°. So when My character rotates, the face I hit with the ray becomes the one I was standing on before, causing me to rotate back and forth between 2 faces.

Here is a beautiful scheme of what I mean:

I tried using SphereCast instead of Raycast, but it didn't work. How can I limit the back and forth rotations ?

Here is the script for the raycast:

void Update(){

...

 if (Physics.Raycast(rayStart, GetDirection(IsGrounded, GravityChangeHitPoint), out RaycastHit hit, Mathf.Infinity, layerMask))
    {
        Debug.DrawLine(transform.position, hit.point,Color.red);
        Quaternion matchSurface;

        if (GravityChangeHitPoint == GravityCenter.position)
            matchSurface = PlayerLookRotation(transform.forward, hit.normal);
        else
            matchSurface = PlayerLookRotation(transform.forward, -GetDirection(IsGrounded, GravityChangeHitPoint));
        

        Quaternion cam = Quaternion.Euler(0, Input.GetAxis("Mouse X") * sensitivityX, 0);

        if (!IsGrounded)
            transform.rotation = Quaternion.Lerp(transform.rotation, matchSurface, Time.deltaTime * damping) * cam;                            
        else
            transform.rotation = matchSurface * cam;
    }  
}

Quaternion PlayerLookRotation(Vector3 approximateForward, Vector3 exactUp)
{
    Quaternion zToUp = Quaternion.LookRotation(exactUp, -approximateForward);
    Quaternion yToZ = Quaternion.Euler(90, 0, 0);

    return zToUp * yToZ;
}

Answer 1

Using orthogonal distance to/from the cube edge as input parameter t, you may want to linearly interpolate your character's local euler angle around a single axis, or else spherically interpolate your character's rotation. That is, your final rotation will be smoothly interpolated between 0.0 and 1.0 of the rotation between your first plane's angle and your second plane's angle.

The problem with both of these approaches, is that the result could still look unnatural on a sharp-edged cube. In that case I would smooth the geometry at the edges of the cube mesh as well, but this depends very much on your requirements.