Here are some tips on camera rotation (mouselook). After naiively implementing a camera class from scratch, I found I had to do a few additional tweaks for good rotational behavior:
Reset mouse coordinates to center-of-screen on each frame, so that mouse never gets caught on the screen borders
Maintain the camera's "up" vector (disallow roll) and recompute the "sideways" vector
Disallow looking up past the vertical +y axis, or down past the -y axis (too far up/down)
Get the order of rotations correct (up/down first, then left/right)
Renormalize the "up", "aim", and "sideways" vectors each frame
Hopefully you can use some of this code to your advantage:
const int mouseDeltaX = mouseAxisX * (input.GetMouseX() - int(app.GetWidth()/2));
const int mouseDeltaY = -mouseAxisY * (input.GetMouseY() - int(app.GetHeight()/2)); // mouse y-offsets are upside-down!
// HACK: reset the cursor pos.:
float lookRightRads = mouseDeltaX * CAMERA_ANGULAR_SPEED_DEG * DEG_TO_RAD;
float lookUpRads = mouseDeltaY * CAMERA_ANGULAR_SPEED_DEG * DEG_TO_RAD;
// Limit the aim vector in such a way that the 'up' vector never drops below the horizon:
static const float zenithMinDeclination = DEG_TO_RAD * MIN_UPWARDS_TILT_DEG;
static const float zenithMaxDeclination = DEG_TO_RAD * (180.0f - MIN_UPWARDS_TILT_DEG);
const float currentDeclination = std::acosf(camera.aim_.y_); ///< declination from vertical y-axis
const float requestedDeclination = currentDeclination - lookUpRads;
// Clamp the up/down rotation to at most the min/max zenith:
if(requestedDeclination < zenithMinDeclination)
lookUpRads = currentDeclination - zenithMinDeclination;
else if(requestedDeclination > zenithMaxDeclination)
lookUpRads = currentDeclination - zenithMaxDeclination;
// Rotate both the "aim" vector and the "up" vector ccw by
// lookUpRads radians within their common plane -- which should
// also contain the y-axis: (i.e. no diagonal tilt allowed!)
ASSERT_ORTHONORMAL(camera.aim_, camera.up_, camera.right_);
// Rotate both the "aim" and the "up" vector ccw about the vertical y-axis:
// (again, this keeps the y-axis in their common plane, and disallows diagonal tilt)
mouseAxisX and mouseAxisY are defined to be +/-1, depending on whether you want the x or y axis mouselook inverted. Usually games offer this option at least for the vertical axis.
MIN_UPWARDS_TILT_DEG is defined to be 1.0 degrees (so the viewer is allowed to look from -89 deg downwards to +89 degrees upwards, which looks quite convincingly like a full 180 degree vertical range -- the missing 2 degrees at the extremes are quite negligible).
camera.aim_, camera.right_, and camera.up_ are of course 3D vectors, and the rotateAboutAxis() method you can cobble together from wikipedia and any number of online sources. Y_AXIS is a fixed constant (0,1,0) vector.
ASSERT_ORTHONORMAL() is a debug-mode-only sanity check, which never gets compiled in optimized/release mode.
Apologies in advance for the C-style code... then again, here you are taking advice from a guy named Mediocritus! ;^)