Why does the object rotate to a random direction when I start dragging?

Question

I'm trying to do drag to rotate sort of thing, so far the code is fine. However when starting to drag (left clicking first), the object rotate to a random direction first which looks odd.

here is my code :

public Camera cam;
public float smoothAngle, dragPower = .3f;
public bool isRadial = true;
float angle, angleOld, angleDelta, angleRef;
Vector2 mousePosOld, mousePosDelta = Vector2.zero;

void Update()
{
  Vector2 mousePos = cam.ScreenToWorldPoint(Input.mousePosition);
  if (Input.GetMouseButton(0))
  {
    mousePosDelta = mousePos - mousePosOld; 
    float rotX = (mousePosDelta.magnitude / Time.deltaTime) 
                  * dragPower 
                  * -Mathf.Sign(mousePos.y) 
                  * Mathf.Sign(Vector2.Dot(Vector2.right, mousePosDelta));

    transform.Rotate(Vector3.forward, rotX);

    mousePosOld = mousePos;
    angleDelta = Mathf.DeltaAngle(angleOld, transform.localEulerAngles.z);
    angleOld = transform.localEulerAngles.z;
  }
  else
  {
    transform.RotateAround(transform.position, transform.forward, angleDelta);
    angleDelta = Mathf.SmoothDampAngle(angleDelta, 0, ref angleRef, smoothAngle);
  }
}

this is the effect :

Answer 1

It looks like there is garbage in your mousePosOld variable when you first detect a click (through if (Input.GetMouseButton(0))): it seems you use an uninitialized value for it (most likely (0, 0)) for the first click, or the value you got from the last time you had mouse-down.

Assuming you'll not get a mouse click event on the first frame, you could update the mousePosOld before leaving the function.

It also looks like you could keep mousePosDelta local to the function; here is an example:

public Camera cam;
public float smoothAngle, dragPower = .3f;
public bool isRadial = true;
float angle, angleOld, angleDelta, angleRef;
Vector2 mousePosOld;

void Update()
{
  Vector2 mousePos = cam.ScreenToWorldPoint(Input.mousePosition);
  if (Input.GetMouseButton(0))
  {
    Vector2 mousePosDelta = mousePos - mousePosOld; 
    float rotX = (mousePosDelta.magnitude / Time.deltaTime) 
                  * dragPower 
                  * -Mathf.Sign(mousePos.y) 
                  * Mathf.Sign(Vector2.Dot(Vector2.right, mousePosDelta));

    transform.Rotate(Vector3.forward, rotX);

    angleDelta = Mathf.DeltaAngle(angleOld, transform.localEulerAngles.z);
    angleOld = transform.localEulerAngles.z;
  }
  else
  {
    transform.RotateAround(transform.position, transform.forward, angleDelta);
    angleDelta = Mathf.SmoothDampAngle(angleDelta, 0, ref angleRef, smoothAngle);
  }

  mousePosOld = mousePos;
}