I am trying to change the target of the smooth follow at runtime, but there is a glitch in the camera since the targets are different sizes. I tried using lerp and smooth damp both gave the same result. How do I overcome the jerk in camera motion?
From the sound of it the error is probably not due to the size of the targets (as that would not modify their position, only their volume). Try using a larger time scale with smoothdamp, perhaps only used during a change over period between targets.
Another option would be to use slerp instead during the switch over; to get the slower acceleration. Without seeing your actual code, it is difficult to see the exact problems (and unfortunately I am not in front of my development machine at this precise time).
Side note: What you MAY want to program in is a "springy-ness" factor into the camera tracking code. I'd look into implementing this, and that code would probably solve your problem (as this sounds like a "camera snap" issue, rather than a "glitch" per say).