Think of the physics of holding a camera. Is it a heavy or a light-weight camera? A heavy one will require more energy to move and rotate, so it will naturally move slower, and thus smoother.
Assuming you have a hand-held camera, you are standing somewhere and pointing it at some position. Then there are two things you can do: you can start walking or running with the camera in some direction, while keeping the orientation of the camera, or you can stand still and rotate the camera until it points to something else. So there's translation of the camera's origin, and a rotation of the camera's pointing vector.
Both of these parameters will not change instantly, and as you have discovered, also not with a linear interpolation. The reason linear interpolation is bad is that during the motion it is too smooth to be natural, and right at the beginning and end of the motion there is a discontinuity. Instead of linear interpolation, you want to use a function that has a more natural smoothness. What you want to do is limit the acceleration of the change in the parameter, or to put it more mathematically, limit its second derivative.
Instead of trying to model it as a mathematical function, I would keep track of the current value of the position and rotation parameters, and the speed at which they are currently changing. Then, before rendering the next frame, you want to calculate a new speed, and then update the current value using the speed times the duration of the frame. To update the speed, just think of what you would do yourself when walking around with a camera: you will have some maximum speed you cannot exceed, and while you are not yet at the right position, you will increase the speed to do to the desired position faster (until you hit the maximum speed), and if you are almost at the target position you will decrease the speed. The latter might be hard to do correctly, but that's fine, a little overshoot might make it look more natural.
Add some noise to the speed variable at every time step to avoid having a camera that looks perfectly still. This will result in Brownian motion. The strength of it can be controlled with a constant that you tweak until you get the desired result. Or you can make that a variable parameter as well, for example increase it to simulate the person holding the camera being scared. The feedback loop mentioned above will ensure the camera will, on average, stay centered on the target.
