Quaternion based camera pitch freaking out, and Z-axis drift

Question

This is the code I use to control my (first person) camera's movement and rotation. Translation successfully keeps orientation in mind, so that up and left and such are always in the expected directions, and 'mouselook' controlling rotation around the Y (yaw) and X (pitch) axes starts out working fine as well.

My problems are two-fold;

First, rotating around the Y and X axes slowly builds up rotational drift around the Z axis. Now what I think I need to do to correct this is to keep track of an additional 'up' vector, and to rotate the quaternion so that the camera's up direction always coincides with this up vector (which would only change when intended, instead of by drift). But I have no idea how to go about doing this.

The second problem is more clearly a bug. When rotating around the Y (yaw) axis, the rotation around the X (pitch) axis starts to freak out. At Y 180 degrees, the up and down motion of the camera's pitching are precisely inverted. And at 90 degree Y angles from the starting orientation, it's as if pitch has become roll instead, rotating around the Z axis.

So, how do I 'roll up', and what's going on with the pitching?

glm::quat mOrientationQuaternion;
glm::vec3 mPositionVector;
glm::mat4 mViewMatrix;

/*
 *  Takes a translation value for each axis, turns it into a vector, then
 *  rotates that vector by the current orientation in the quaternion, before
 *  adding the result to the position vector.
 */
void Camera::translate(float x, float y, float z)
{
    glm::vec3 translation(x, y, z);
    translation = translation * mOrientationQuaternion;
    mPositionVector += translation;
}

/*
 *  Takes three angles in degrees, and creates a quaternion via an
 *  intermediate vector to represent their rotations. Then the orientation
 *  quaternion is multiplied by this new rotation quaternion to get the new
 *  orientation.
 */
void Camera::rotate(float x, float y, float z)
{
    glm::vec3 angles(glm::radians(-x), glm::radians(y), glm::radians(z));
    glm::quat rotation(angles);
    mOrientationQuaternion = glm::normalize(mOrientationQuaternion * rotation);
}

/*
 *  Creates the view matrix from the orientation quaternion, then translates
 *  it by the position vector.
 */
void Camera::buildViewMatrix()
{
    mViewMatrix = glm::mat4_cast(mOrientationQuaternion);
    mViewMatrix = glm::translate(mViewMatrix, mPositionVector);
}

Answer 1

Firstly, I would recommend working in radians, not degrees. Whilst the GLM library can work with degrees, it was designed with radians in minds(this is a very minor issue though, so work with what you feel comfortable in).

Secondly, If you wish to use quaternions to store data as orientation, then you must understand conceptually, what a quaternion is. Forget about the mathematics for a moment, because that is less important at this point, and to be honest, if you're using glm, you only need to know how to use it, and not how it works under the hood.

A quaternion represents a rotation around an axis, in three dimensions. I wouldn't recommend trying to use it for anything else, such as for translation, as that will just confuse the issue, and make your code difficult to understand six months from now.

Consider the following code:

glm::vec3 mPositionVector;
glm::quat mOrientation;
glm::vec3 mAngularVelocityRadS; //this is actually a set of Euler angles

This has all the information you will ever need to compute a stable view matrix.

If you wish to rotate the camera in any direction, simply do this:

glm::quat rotateCam(glm::quat orientation, glm::quat angularVelocityS, float deltaTime) {
    //first we compute the new orientation after 1 second.
    glm::quat rot1s = angularVelocityS * orientation;
    // Now we know what the rotation would be after 1 second
    // We can now perform spherical linear interpolation
    // to compute something in between.
    return glm::normalize(glm::mix(orientation, rot1s, deltaTime));
}

We now have a very nice general function for rotations, but how do we use that?

void camera::update(float dt) {
    glm::quat rotation = glm::quat(mAngularVelocityRadS);
    mOrientation = rotateCam(mOrientation, mAngularVelocityRadS, dt);
    // Now we have our orientation, we can compute the view matrix
    mViewMatrix = glm::mat4_cast(glm::inverse(mOrientationQuaternion));
    mViewMatrix = glm::translate(mViewMatrix, mPositionVector);
}

if you wish to rotate the camera by a fixed amount, then just multiply the orientation by the rotation, without slerp needed:

mOrientation = rotation * mOrientation;

If you want to reset the orientation:

mOrientation = glm::quat(glm::vec3(.0f,.0f,.0f));

Answer 2

I've managed to fix the pitch-freakout issue, but at this point this is pretty much 'Programming by Coincidence', as I don't understand the math well enough to know -why- my changes fix the problem.

First, I get the quaternion's inverse before turning it into a view matrix. Now intuitively I'd guess this has to do with the difference between 'rotating the camera' (which doesn't actually exist), and 'rotating the entire world in the opposite direction', but I might be completely off-base here;

void Camera::buildViewMatrix()
{
    mViewMatrix = glm::mat4_cast(glm::inverse(mOrientationQuaternion));
    mViewMatrix = glm::translate(mViewMatrix, mPositionVector);
}

Doing so messes up translation, so next I change translation to translate by the inverse of the orientation quaternion. At this point I'm completely lost as to the mathematical how-and-why;

void Camera::translate(float x, float y, float z)
{
    glm::vec3 translation(x, y, z);
    translation = translation * glm::inverse(mOrientationQuaternion);
    mPositionVector += translation;
}

And finally I had to flip the x and y axes' signs during rotation;

void Camera::rotate(float x, float y, float z)
{
    glm::vec3 angles(glm::radians(x), glm::radians(-y), glm::radians(z));
    glm::quat rotation(angles);
    mOrientationQuaternion = glm::normalize(mOrientationQuaternion * rotation);
}

The final result is a camera that pitches and yaws by mouse motion, and that translates in all directions intuitively.

Because of how much guesswork and experimentation this has been, I have no idea how efficient this code is, so it might be doing incredibly wasteful things that'd make any mathematician groan. Any comments on whether taking the inverse twice (or normalizing after every rotation) is a good or bad idea would be most welcome. :-)

Having fixed one issue, I still need to figure out how to get rid of the axial drift. For now I still assume this is something I can handle with a dedicated Up vector and a corrective z rotation back to Up after every rotation around the other axes.