Camera movement issue with pitch and yaw angles

Question

I'm trying to implement a simple first person camera using pitch and yaw angles (no quaternions, I want a simple camera for debugging purposes). An Input class processes mouse input and translate the x and y position in the window into angles. Here's the code:

#define SENSITIVITY 0.01f

void Input::ProcessMouseInput(int x, int y, Camera &camera)
{
    /* pitch and yaw angles */
    static float pitch = 0;
    static float yaw = 0;

    /* initialize initial mouse position */
    static int prev_x = x;
    static int prev_y = y;

    int offset_x = x - prev_x;
    int offset_y = y - prev_y;

    pitch += offset_y * SENSITIVITY;
    yaw += offset_x * SENSITIVITY;

    /* compute transformation matrix */
    XMMATRIX pitchTransform = XMMatrixRotationX(pitch);
    XMMATRIX yawTransform = XMMatrixRotationY(yaw);
    XMMATRIX transform = XMMatrixMultiply(pitchTransform, yawTransform);    // fps-like  
    //XMMATRIX transform = XMMatrixMultiply(yawTransform, pitchTransform);  // airplane-like

    /* transform camera direction - rotate camera */
    XMVECTOR oldDirection = XMVectorSet(0.0, 0.0, 1.0, 1.0);
    XMVECTOR newCameraDirection_ = XMVector3Transform(oldDirection, transform);
    XMMATRIX oldView = XMLoadFloat4x4(&camera.GetInitialViewMatrix());
    XMVECTOR det = XMMatrixDeterminant(oldView);
    XMMATRIX inverse = XMMatrixInverse(&det, oldView);
    XMVECTOR v = XMVector3TransformNormal(newCameraDirection_, inverse);
    XMFLOAT3 newCameraDirection;
    XMStoreFloat3(&newCameraDirection, v);
    camera.SetDirection(newCameraDirection);

    /* store current mouse position */
    prev_x = x;
    prev_y = y;
}

What I do is I calculate the direction vector in the view space and transform it with pitch/yaw rotations. Then I tranform the resulting vector in world space coordinates and update the camera direction. All works fine if the camera space x-z plane initial position is parallel to the world space x-z plane. If the camera is initially tilted downwards or upwards the movement is off (moving along the yaw direction causes a wobbling and the movement is incorrect). I can't find the problem as I think the calculations are correct (but they obviously aren't).

EDIT

what I mean with odd behaviour is that when I create a camera that has starting position with a pitch wich is not zero I get mixed movements (pitch when yawing and yawing when pitching)

Answer 1

This block of code here:

XMVECTOR det = XMMatrixDeterminant(oldView);
XMMATRIX inverse = XMMatrixInverse(&det, oldView);
XMVECTOR v = XMVector3TransformNormal(newCameraDirection_, inverse);
XMFLOAT3 newCameraDirection;
XMStoreFloat3(&newCameraDirection, v);
camera.SetDirection(newCameraDirection);

says "always rotate with regard to the initial camera orientation." That means if the initial camera orientation was inclined downwards, all future yaw should happen in that inclined horizontal plane - leading to the wobble you describe.

If you don't want that, I'd recommend not using the camera's initial orientation matrix as a basis for future updates. Instead, initialize your pitch & yaw to match the initial rotation so you start from the right place, and rotate with regard to the identity matrix instead.