Quats are very similar to an angle-axis representation, nothing more.

 So if you apply a quat, you imply that the result is the input rotated against the quat's axis by the specified angle.

 What does that tell us? 

It tells me this: that the quat's axis must be perpendicular to the camera's lookat axis in order to not rotate the camera against this look at ray. 

If the axes are not perpendicular, there will be a roll component taking its toll, and hence the OPs problem.

If required, I could provide further details (a 3D rep maybe?).