Smaller data helps a lot with performance which becomes critical when you have dozens or hundreds of characters with multiple blended animations.
There can also be several advantages to using quaternions when interpolating between frames of an animation. Matrices can theoretically fail to represent certain series of transformations (gimbal lock) and linear interpolation of rotational matrices gives you a slightly incorrect values compared to spherical interpolation of quaternions.
In short, use quaternions and offset vectors. You can reconstruct the final transformation matrix easily enough on the GPU during skinning.
This a gets to be a much longer conversation once animation compression and other high-end techniques are tossed in, but you still typically stick with quaternions.