How do I change quaternion rotation to use the local camera axis rather than the world axis in DirectX 11?

Question

I'm currently trying to rotate the camera around its local axis based on keyboard/mouse input and the code I currently have uses DirectXMath and works nicely, however it is using the world axis to rotate around rather than the cameras local axis. Because of this, some of the rotations are not as expected and causes issues as the camera rotates. For example, when we tilt our camera, the Y axis will change and we will want to rotate around another axis to get our expected results.

What am I doing wrong in the code or what do I need to change in order to rotate around its local axis?

vector.x, vector.y, vector.z (The vector to rotate around, i.e. (1.0f, 0.0f, 0.0f))

//define our camera matrix
XMFLOAT4X4 cameraMatrix;

//position, lookat, up values for the camera
XMFLOAT3 position;
XMFLOAT3 up;
XMFLOAT3 lookat;

void Camera::rotate(XMFLOAT3 vector, float theta) {
    XMStoreFloat4x4(&cameraMatrix, XMMatrixIdentity());

//set our view quaternion to our current camera's lookat position
XMVECTOR viewQuaternion = XMQuaternionIdentity();
viewQuaternion = XMVectorSet(lookat.x, lookat.y, lookat.z, 0.0f);

//set the rotation vector based on our parameter, i.e (1.0f, 0.0f, 0.0f)
//to rotate around the x axis
XMVECTOR rotationVector = XMVectorSet(vector.x, vector.y, vector.z, 0.0f);
//create a rotation quaternion to rotate around our vector, with a specified angle, theta
XMVECTOR rotationQuaternion = XMVectorSet(
XMVectorGetX(rotationVector) * sin(theta / 2),
XMVectorGetY(rotationVector) * sin(theta / 2),
XMVectorGetZ(rotationVector) * sin(theta / 2),
cos(theta / 2));

//get our rotation quaternion inverse
XMVECTOR rotationInverse = XMQuaternionInverse(rotationQuaternion);

//new view quaternion = [ newView = ROTATION * VIEW * INVERSE ROTATION ]
//multiply our rotation quaternion with our view quaternion
XMVECTOR newViewQuaternion = XMQuaternionMultiply(rotationQuaternion, viewQuaternion);

//multiply the result of our calculation above with the inverse rotation
//to get our new view values
newViewQuaternion = XMQuaternionMultiply(newViewQuaternion, rotationInverse);

//take the new lookat values from our newViewQuaternion and put them into the camera
lookat = XMFLOAT3(XMVectorGetX(newViewQuaternion), XMVectorGetY(newViewQuaternion), XMVectorGetZ(newViewQuaternion));

//build our camera matrix using XMMatrixLookAtLH
XMStoreFloat4x4(&cameraMatrix, XMMatrixLookAtLH(
    XMVectorSet(position.x, position.y, position.z, 0.0f),
    XMVectorSet(lookat.x, lookat.y, lookat.z, 0.0f),
    XMVectorSet(up.x, up.y, up.z, 0.0f)));
}

The view matrix is then set

//store our camera's matrix inside the view matrix
XMStoreFloat4x4(&_view, camera->getCameraMatrix() );

-

Edit:

I have tried an alternative solution without using quaternions, and it seems I can get the camera to rotate correctly around its own axis, however the camera's lookat values now never change and after I have stopped using the mouse/keyboard, it snaps back to its original position.

void Camera::update(float delta) {
    XMStoreFloat4x4(&cameraMatrix, XMMatrixIdentity());

//do we have a rotation?
//this is set as we try to rotate, around a current axis such as
//(1.0f, 0.0f, 0.0f)
if (rotationVector.x != 0.0f || rotationVector.y != 0.0f || rotationVector.z != 0.0f) {

    //yes, we have an axis to rotate around

//create our axis vector to rotate around
    XMVECTOR axisVector = XMVectorSet(rotationVector.x, rotationVector.y, rotationVector.z, 0.0f);

//create our rotation matrix using XMMatrixRotationAxis, and rotate around this axis with a specified angle theta
    XMMATRIX rotationMatrix = XMMatrixRotationAxis(axisVector, 2.0 * delta);

//create our camera's view matrix
    XMMATRIX viewMatrix = XMMatrixLookAtLH(
        XMVectorSet(position.x, position.y, position.z, 0.0f),
        XMVectorSet(lookat.x, lookat.y, lookat.z, 0.0f),
        XMVectorSet(up.x, up.y, up.z, 0.0f));

//multiply our camera's view matrix by the rotation matrix
//make sure the rotation is on the right to ensure local axis rotation
    XMMATRIX finalCameraMatrix = viewMatrix * rotationMatrix;


/* this piece of code allows the camera to correctly rotate and it doesn't
snap back to its original position, as the lookat coordinates are being set
each time. However, this will make the camera rotate around the world axis
rather than the local axis. Which brings us to the same problem we had
with the quaternion rotation */
    //XMVECTOR look = XMVectorSet(lookat.x, lookat.y, lookat.z, 0.0);

    //XMVECTOR finalLook = XMVector3Transform(look, rotationMatrix);

    //lookat.x = XMVectorGetX(finalLook);
    //lookat.y = XMVectorGetY(finalLook);
    //lookat.z = XMVectorGetZ(finalLook);

//finally store the finalCameraMatrix into our camera matrix
    XMStoreFloat4x4(&cameraMatrix, finalCameraMatrix);
} else {

    //no, there is no rotation, don't apply the roation matrix

//no rotation, don't apply the rotation matrix
    XMStoreFloat4x4(&cameraMatrix, XMMatrixLookAtLH(
        XMVectorSet(position.x, position.y, position.z, 0.0f),
        XMVectorSet(lookat.x, lookat.y, lookat.z, 0.0f),
        XMVectorSet(up.x, up.y, up.z, 0.0f)));
}

An example can be seen here: https://i.gyazo.com/f83204389551eff427446e06624b2cf9.mp4

I think I am missing setting the actual lookat value to the new lookat value, but I'm not sure how to calculate the new value, or extract it from the new view matrix (which I have already tried)

Answer 1

Looking at the code, I have a bit of trouble understanding exactly the way it works, but the way I would go about implementing a camera rotation is simply using rotation matrices to rotate the camera local axes. Note: I haven't worked with DirectXMath too much, so in the next code I will use imaginary math functions that I am sure you can then replace with DirectXMath equivalents.

So the way I would go about it is to first have a camera class, something along the lines of:

class Camera
{
 public:

    // Constructors, public methods etc
    void Update();

 private:

    Vector3 _position; // Camera position
    Vector3 _right;    // Camera local right vector
    Vector3 _up;       // Camera local up vector
    Vector3 _look;     // Camera local look vector

    Matrix4x4 _viewMatrix;
}

Of course this is just a simple skeleton of the class. You would probably also want to have other pieces of info such as projection matrix, viewport etc. But these are the members that I consider important for this example.

I will also assume in this example that the input handling will take place in the Update method. So let's see how you could implement the camera rotation you desire inside the Update method. Note: There are many different way in which you could rotate a camera, but I will assume that you would like a FPS rotation style. Even if a new rotation style is required, I think the following example will help.

void Camera::Update()
{
   // Calculate mouse deltas... Not shown here, but can easily be done
   // by keeping track of the previous mouse position and subtracting
   // that from the current position. I assume you are already doing
   // that so I will just be using the mouse delta vector as if it was
   // already initialized. deltaX holds the mouse movement along the X
   // axis and deltaY along the Y axis in screen space.

   // For a FPS camera, deltaX rotates the camera around the GLOBAL Y 
   // axis. This is as if you are rotating your entire body around the 
   // world UP vector. deltaY rotates around the camera's local X axis.
   // This can be associated with tilting your head up and down. 
   float yRotation = deltaX; // * sensitivityValue (optional);
   float xRotation = deltaY; // * sensitivityValye (optional);

   // Build a rotation matrix which rotates the camera around GLOBAL Y
   Matrix4x4 yAxisRotMtx = RotationAxis(yRotation, Vector3(0.0f, 1.0f, 0.0f);

   // Rotating the camera around the Y axis can be accomplished by 
   // rotating the camera local axes using the rotation matrix we've just
   // built.
   _right = yAxisRotMtx.MultiplyVector(_right);
   _up = yAxisRotMtx.MultiplyVector(_up);
   _look = yAxisRotMtx.MultiplyVector(_look);

   // At this point we have rotated our body around in the world based 
   // on the horizontal mouse movement. The next step is to tilt our
   // head based on the vertical mouse movement. In order to do this, we
   // need to construct a rotation matrix which rotates the camera local
   // vectors around the camera's local right vector.
   // NOTE: We only need to rotate the camera look and up vectors.
   //       Rotating the right vector around itself will have no effect.
   Matrix4x4 camRightRotMtx = RotationAxis(xRotation, _right);
   _up = camRightRotMtx.MultiplyVector(_up);
   _look = camRightRotMtx.MultiplyVector(_look);

   // At this point we have achieved the desired rotation effect and we
   // are ready to rebuild our view matrix. The view matrix needs to be
   // rebuilt because we have modified the camera local axes and this
   // is what the view matrix essentially contains (aside from positional
   // information). There are 2 ways in which we could calculate the view
   // matrix: one which is a bit slower, and the other one which is 
   // probably a bit trickier to understand but it is faster. We will
   // pretend that a MACRO will be used which allows you
   // to switch between the 2 so we will implement both of them.
   #if defined(USE_SLOW_VIEWMTX_CALCULATION)
   // Slow method. Use the camera local vectors and its position to
   // calculate a world matrix which essentially describes the camera's
   // position and orientation in the 3D world. Inverting this matrix 
   // will give you the view matrix. The inversion is needed because 
   // the view matrix is supposed to transform points from world space
   // to camera local space.
   Matrix4x4 camWorldMtx;
   camWorldMtx.MakeIdentity();

   // Each row inside the matrix holds a camera local axis. The fourth
   // row holds the camera position.
   camWorldMtx._00 = _right.x; camWorldMtx._01 = _right.y; camWorldMtx.02 = _right.z;
   camWorldMtx._10 = _up.x; camWorldMtx._11 = _up.y; camWorldMtx.12 = _up.z;
   camWorldMtx._20 = _look.x; camWorldMtx._21 = _look.y; camWorldMtx.22 = _look.z;
   camWorldMtx._30 = _position.x; camWorldMtx._31 = _position.y; camWorldMtx.32 = _position.z;

   // The inverse of the matrix we calculated is the view matrix
   _viewMatrix = camWorldMtx.GetInverse();
   #else
   // Faster method. Requires a bit of understanding as to what the view
   // matrix is trying to achieve and that discussion can get a bit
   // lengthy, but the idea is that if we store the camera local vectors
   // in the columns (not rows as was the case in the first calculation
   // method) we are essentially creating a matrix which rotates points
   // out of world space and into camera local space. Rotation is half
   // the journey. We must also calculate the fourth row of the matrix. 
   // This is not a simple case of just negating the camera position
   // coords because when a vertex is transformed by the view matrix, by
   // the time it reaches the fourth row, the first 3 rows have already
   // affected the vertex (brought it in camera local space). So we need
   // to ransform the position of the camera in the same space before
   // subtracting it from the vertex. This is done by transforming the 
   // camera position via the 3x3 portion of the view matrix, negating
   // the result and storing it. Sorry if this creates more confusion
   // than clarity :) 
   _viewMatrix._00 = _right.x;
   _viewMatrix._10 = _right.y;
   _viewMatrix._20 = _right.z;

   _viewMatrix._01 = _up.x;
   _viewMatrix._11 = _up.y;
   _viewMatrix._21 = _up.z;

   _viewMatrix._02 = _look.x;
   _viewMatrix._12 = _look.y;
   _viewMatrix._22 = _look.z;

   // Now position. Performing the dot product for each position
   // component with the corresponding camera axis has the same
   // effect as multiplying the position with the view matrix. We
   // negate the result because we want to bring points relative to
   // the camera's origin (i.e. cancel out any positional information).
   _viewMatrix._30 = -Vector3.Dot(_position, _right);
   _viewMatrix._31 = -Vector3.Dot(_position, _up);
   _viewMatrix._32 = -Vector3.Dot(_position, _look);
   #endif
}

There is an optimization which can be performed. For example, the view matrix will most likely be needed only when finally rendering the scene. Camera manipulation happens before that. So instead of calculating the camera view matrix inside the Update method, you could just use a _viewMtxIsDirty flag to mark the view matrix as dirty. Then, when it comes time to render the scene, you might call a function from your camera class, something like GetViewMatrix(). This function could be implemented as such:

Matrix4x4 Camera::GetViewMatrix()
{
    if(_viewMtxIsDirty) { RebuildViewMatrix(); _viewMtxIsDirty = false; }
    return _viewMtxIsDirty;
}

The RebuildViewMatrix() method would implement one of the discussed calculation methods.

I saw you were trying to use quaternions to achieve rotation and I believe it can work, but to be honest, quaternions can get confusing sometimes (at least for me :D ). Using rotation matrices seems much more intuitive to me. That is why I chose to go down this road. Hope this helps.