Unity clamping local rotation on LookAt issues

Question

I'm new to game development, and I've been trying to code a "smooth camera" rotation that's limited within the camera's local rotation, using a smoothed cube and transform.lookAt. The code works fine when parent is rotated straight on, but begins to flicker when the parent is rotated -90 degrees, and becomes completely reversed when rotated -180 and +180.

Here's a video of the issue in play: https://streamable.com/sqzyb4

and here's the code:

[Header("References")]
public Camera cam;
public Transform cursorDelay;
public Transform cameraHandler;

[Header("Options")]
public float smoothSpeed;
public float zOffset;

[Header("Camera Restrictions")]
public float maxCameraRotationX;
public float maxCameraRotationY;

[Header("Sensitivity/Reach")]
public float cameraXSensitivity;
public float cameraYSensitivity;

private float camX;
private float camY;


private void Start()
{
    Cursor.lockState = CursorLockMode.Confined;
}

private void Update()
{
    //Takes mouse position and lerps cursorDelay with it, smoothing the camera movement.

    Vector3 cursorPos = Input.mousePosition;
    cursorPos.z = zOffset;
    Vector3 smoothedPos = Vector3.Lerp(cursorDelay.transform.position, cam.ScreenToWorldPoint(cursorPos), smoothSpeed * Time.deltaTime);
    cursorDelay.transform.position = smoothedPos;

    cam.transform.LookAt(smoothedPos);

    CameraRestrictions();

}

//Restricts Camera's local rotation
private void CameraRestrictions()
{
    camX = cam.transform.localRotation.x;
    camY = cam.transform.localRotation.y;


    camX += Mathf.Clamp(camX, -maxCameraRotationX, maxCameraRotationX);
    camY += Mathf.Clamp(camY, -maxCameraRotationY, maxCameraRotationY);

    cam.transform.localRotation = Quaternion.Euler(camX * cameraXSensitivity, camY * cameraYSensitivity, 0f);

}

If anyone knows a fix or something I did wrong, please let me know.