Based on you comments, it seems that you're storing the orientation of the object as a set of Euler angles, and in/decrementing the angles when the player rotates the object. That is, you have something like this pseudocode:
// in player input handling:
if (axis == AXIS_X) object.angleX += dir;
else if (axis == AXIS_Y) object.angleY += dir;
else if (axis == AXIS_Z) object.angleZ += dir;
// in physics update and/or draw code:
matrix = eulerAnglesToMatrix(object.angleX, object.angleY, object.angleZ);
As Charles Beattie notes, because rotations don't commute, this won't work as expected unless the player rotates the object in the same order in which the eulerAnglesToMatrix()
applies the rotations.
In particular, consider the following sequence of rotations:
- rotate object by x degrees around the X axis;
- rotate object by y degrees around the Y axis;
- rotate object by −x degrees around the X axis;
- rotate object by −y degrees around the Y axis.
In the naïve Euler angle representation, as implemented in the pseudocode above, these rotations will cancel out and the object will return to its original orientation. In the real world, this does not happen — if you don't believe me, grab a six-sided die or a Rubik's cube, let x = y = 90°, and try it out yourself!
The solution, as you note in your own answer, is to store the orientation of the object as a rotation matrix (or a quaternion), and update that matrix based on user input. That is, instead of the pseudocode above, you'd do something like this:
// in player input handling:
if (axis == AXIS_X) object.orientation *= eulerAnglesToMatrix(dir, 0, 0);
else if (axis == AXIS_Y) object.orientation *= eulerAnglesToMatrix(0, dir, 0);
else if (axis == AXIS_Z) object.orientation *= eulerAnglesToMatrix(0, 0, dir);
// in physics update and/or draw code:
matrix = object.orientation; // already in matrix form!
(Technically, since any rotation matrix or quaternion can be represented as a set of Euler angles, it is possible to use them to store the orientation of the object. But the physically correct rule for combining two sequential rotations, each represented as Euler angles, into a single rotation is rather complicated, and essentially amounts to converting the rotations into matrices / quaternions, multiplying them, and then converting the result back into Euler angles.)