I'm trying to implement a free-look third person camera (using glm). I know that the general transformation is
newCameraPosition = translate(lookatPoint) * rotation(angle,axis) * translate(-lookatPoint);
So I need to find the axis to rotate around, with the constraint that I give the algorithm an axis in camera space to rotate around. Sounds confusing.
For example, let's take a camera that points up (0,1,0), is positioned in the XY plane at (-1,0,0) and is looking at the origin (0,0,0). So it is looking "along the x-axis".
Now I want to rotate the camera around the "camera-space-x-axis" (1,0,0). In words: The axis that "points to the right" when looking through the camera. Simply rotating around (1,0,0) doesn't work, since it will rotate the camera around the world-space x-axis. From the point of the camera however, the x-axis is in fact the z-axis in world space (0,0,1);
How do I go from the given camera-space axis to the world-space axis? I've tried just applying the inverse transform of the current camera transform to the given camera-space axis using
glm::vec3 rotateAxis = swizzle<X,Y,Z>(glm::inverse(GetCameraTransform()) * cameraSpaceAxis);
where cameraSpaceAxis.w = 0;
but that produces wrong results, e.g. if I give the algorithm the axis (1,0,0) to rotate around, continuously rotation around it in 1 degree steps just creates a wobbly circular motion around y and x axis;