Rotation around arbitrary vector (using quaternions?)

Question

I currently have a camera which orbits a specific target object, always looking at it.

The user can drag the mouse to move the camera left/right/up/down and the camera will move over the surface of a sphere of fixed radius (note, I clamp the vertical angle to ± 89°).

Currently it looks something like this...

// Store position of camera for future reference
private Vector2 rotation = new Vector2(120, 25);

// Was intended to be the vector from planet to target. Only works with `up`
private rotationalPole = Vector3.up;

if (isDragging) {
    rotation += new Vector2(
        Input.GetAxis("Mouse X") * xSpeed * radius * 0.02f,
        -Input.GetAxis("Mouse Y") * ySpeed * 0.02f);

}

//target is the GameObject to center on...

var pos = target.transform.position;

camObj.transform.position = pos + (Vector3.back * actualRadius);
camObj.transform.LookAt(pos, rotationalPole);
camObj.transform.RotateAround(pos, Vector3.right, rotation.y);
camObj.transform.RotateAround(pos, rotationalPole, rotation.x);

This works fine when up is in the y direction.

Now, however, instead of an object on the ground, the target is something in orbit around a planet (currently on an equatorial orbit but hopefully on an arbitrary one in future).

I want the camera to behave the same as before, but with up being the vector from the center of the planet through the target. Nominally forward will be the direction of orbit (but since the user can spin through 360 "horizontally", that's less important).

Currently it looks like this...

Basically, I'd like to rotate the image above counter-clockwise by 90° (which I believe I can do by playing with the camera transform's up and right vectors) but also that my arbitrary rotation honors the new orientation.

Some research shows that I need to use quaternions but while my mental model for basic trig is fine, I can't picture how to use quaternions correctly in this situation.

Following on from @Jon's answer below...

I've now got the following.

var planetToTarget = (target.transform.position - planet.transform.position).normalized;
// "Reset" the camera before I do any transforms
this.transform.position = target.transform.position - tgtMovement * actualRadius;
this.transform.up = Vector3.up;
this.transform.LookAt(target.transform.position);

var camToTarget = (transform.position - target.transform.position).normalized;
Debug.Log(string.Format("targetMotion = {0}, planetToTarget: {1}, camToTarget: {2}", tgtMovement, planetToTarget, camToTarget));

// Line that's not working...
this.transform.Rotate(camToTarget, -90);

// To be added in when the above is working...  
//this.transform.Rotate(Vector3.Cross(camToTarget, planetToTarget), rotation.y);
//this.transform.Rotate(planetToTarget, rotation.x);
//this.transform.up = planetToTarget;

The uncommented code (excluding the last line) keeps the camera behind the target, looking forward...

Initially, I get :

targetMotion = (0.0, 0.0, -1.0), planetToTarget: (-1.0, 0.0, 0.0), camToTarget: (0.0, 0.0, 1.0)

Which generates:

However, as the target moves around the planet, it changes to:

targetMotion = (0.4, 0.0, -0.9), planetToTarget: (-0.9, 0.0, -0.4), camToTarget: (-0.4, 0.0, 0.9)

By which point, the camera is slewing to the side...

So I'm clearly still missing something. I don't appear to be rotating around camToTarget correctly?

Answer 1

Once you calculate the new up, you'll want the lerp from old to new to take a few seconds. Additionally, if CameraToObject and PlanetToObject are coincident, a single plane cannot be defined. If the object is already selected, your vertical constraint will prevent CameraToObject and PlanetToObject from becoming coincident, however, if the two are already coincident when you click the object, or are made coincident by clicking the object, the algorithm will not work. So, if you go to calculate and detect that they are coincident, bump one a tiny bit; this will find an up vector, which allows the camera to begin lerping, which allows the vertical constraint to work.

The cross-product of the normalized PlanetToObject and CameraToObject directions is perpendicular to both vectors and is, by definition, the normal of a plane that contains both. The cross product of the plane's normal (perpendicular to the plane) and CameraToObject (a vector in the plane) is a second vector in the same plane, perpendicular to the first; the net-result is a rotation of CameraToObject 90 degrees around the face normal.

Note: I may or may not have reversed the order of the cross-products, but you only need to worry about it at the very end; the yellow vector should point to the "right of the camera"; invert it if it doesn't.

This was one of the hardest screenshots I've tried to diagram over; I had given up on making it look right but will post it anyway, since I think you'll still get it.

Regarding quats:

Consider the specific wording of your question. "I want to rotate the image counter-clockwise". That's also saying that you want to rotate the camera clock-wise around CameraToObject. In that case, CameraToObject is the Quaternion's vector component, and "90 degrees" is the scalar component. Quaternions twist inputs around their vector component by the angle stored in their scalar component.

You didn't know you've already been using quaternions for matrix math? With unit-axis vector components, the quaternion equation reduces to the appropriate axis-rotation matrices: