Okay, this is the code that works for a FPS camera, except it allows the player to flip it over by going under/over -/+90° rotation on Y-axis:
transformation->rotateObjectFromRight(glm::radians(angleX), glm::tvec3<GLfloat>(0.0f, 1.0f, 0.0f));
transformation->rotateObjectFromLeft(glm::radians(angleY), glm::tvec3<GLfloat>(1.0f, 0.0f, 0.0f));
transformation object holds the data required to construct the view matrix, among others, a quaternion representing the rotation.
The functions called will construct the quaternion with the given data and then multiply it with the rotation quaternion either from the left (relative to global axes) or the right (relative to local axes) side.
angleX and angleY are mouse X and Y offsets representing the angles to rotate around.
The first line yaws the camera around local Y-axis, while the second then pitches it around global X-axis, creating the characteristic FPS camera movement.
Okay. So far so good. This is the code that should limit the pitch to never exceed the +/-90 boundaries:
transformation->rotateObjectFromRight(glm::radians(angleX), glm::tvec3<GLfloat>(0.0f, 1.0f, 0.0f));
glm::tvec3<GLfloat> forwardVector = glm::rotate(glm::inverse(transformation->getRotation()), glm::tvec3<GLfloat>(0.0f, 0.0f, 1.0f));
GLfloat pitchAngle = forwardVector.y * 90.0f;
angleY = glm::clamp(angleY, -(pitchAngle + 90.0f), -(pitchAngle - 90.0f));
transformation->rotateObjectFromLeft(glm::radians(angleY), glm::tvec3<GLfloat>(1.0f, 0.0f, 0.0f));
First and the last line are the same. What happens here?
First I get the unit vector that lies on the Z-axis and rotate it so that it represents the forward vector of the camera. I then take the Y component of that vector (ranging from -1 to 1) and multiply it with 90 to get the degrees. Then I clamp the angleY so that current rotation combined with the new one never exceeds the +/-90° boundary.
If the rotation was about to flip the camera overhead, it would change the new rotation angle so the camera looks straight up. Similar thing for underfoot flipping.
However, the camera acts weird. It's fastest at pitch angles close to 0°, but gets progressively slower as the pitch nears +/-90°. If I move my mouse slower, the camera is faster, if I move it faster, it's slower.
It's as though it's discarding the too large/small angleY values instead of clamping them to the appropriate values (but it's not, and I can confirm via the debugger - all the calculated values look alright and are correct).
What could be amiss here to cause such weird movement?
