How do I rotate a cube around an axis, relative to the camera view?

Question

I want to be able to rotate a cube, either by

• dragging vertically to rotate it around the Y-axis, or
• dragging horizontally to rotate it around the X-axis.

However, when the camera is rotated 90 degrees or more, these directions become wrong relative to the view screen. How can make rotating ‘up’ always mean ‘up’ relative to the camera view?

---

Here's a picture explaining the situation:

Answer 1

The camera up-vector

A property of the camera you could use to solve this problem is the up-vector. This vector indicates which direction in 3D space should correspond with the (upward) vertical axis of your screen. Usually, cameras will use the positive y-axis for this, thus: (0, 1, 0).

---

The scalar product of two vectors

The scalar product of two 3D vectors (x1, y1, z1) and (x2, y2, z2) is defined as:

(x1 . x2) + (y1 . y2) + (z1 . z2)

It has the property that it will be zero for two perpendicular vectors and one for two parallel vectors. Note that this last property assumes your vectors are normalized (=they are unit length).

For intermediate angles, the value of the scalar product increases monotonically with increasing angle between the vectors.

---

Solution: Finding the "most vertical" axis using the scalar product

You can use the above bits of information to solve your problem. In essence you have two local axes of your cube (the X-axis and the Y-axis) and you want to figure out which one aligns most with the vertical axis of your screen. The other axis will naturally align more with the horizontal axis of your screen.

The up-vector of your camera corresponds with this vertical axis. Knowing that the scalar product of two vectors is bigger, the more they align in direction, you can do the following:

1. Normalize the local X-axis and Y-axis vectors of your cube
2. Calculate the scalar product of your up-vector with both these normalized axis vectors
3. If the scalar product with the X-axis is bigger, bind horizontal mouse movement to its rotation and bind vertical mouse movement to the Y-axis rotation. If the scalar product with the Y-axis is bigger, do the inverse.

---

In summary, you can calculate the scalar product of the camera up-vector with the normalized X- and Y-axis vectors of the cube to find which axis best aligns with the vertical axis of your screen.