Quite old thread I see... :) Maybe somebody else will need that.
When I needed similar solution I tried hard with camera parented with the player object and quaternion tricks as you said, etc. and nothing seemed to work as I wanted. Then, I detached my camera from the player and everything became easier.
Just detach your camera and compute some point at the back of your player - this will be the point that you camera wants to get into. For every frame just move your camera by some percent of the distance to the target - this way you get nice, smooth movement.
The same applies to rotation - make some "target" orientation, it may point to the player, or in the same direction the player looks, and for each frame compute some percent of rotation difference to get closer to your target value.
In fact, what you want is some kind of chasing camera. It's not physically connected to the player but it tries to follow him smoothly. When you put your camera as a players child, then it inherits all player transformations, so it starts looking as if it was connected. You get the situation like on some GoPro movies, with camera attached to your helmet and pointing at your face. Then, you try to find some clever math to avoid the effect of your own hierarchy :).
BTW: Remember about your frame rate and numerical integration errors - i.e don't compute like: val = dist * 0.1f
, but val = dist * log(speed,time)