My usual route is to separate the logical and visual parts of the game object:
Colliders can go either at the root, or in a child object separate from the visual child.
This way I can do anything I want to the underlying visual - squash & stretch it to juice up its movement animations, bounce it around, etc. - without impacting the physics simulation or gameplay logic.
It also means we can swap the visual without losing the player controller, by either destroying the old one and instantiating a new visual child object in its place, or by having all the visuals present at all times, but disabling the game object for all but one of them at a time, and just swapping which one is active.
The latter option is good if you cycle appearances between a small-ish set of options frequently, so you're not incurring extra instantiation/destruction overhead and memory allocation constantly. But if you have a very large roster of visuals and each playthrough will only see a few, it might not be more efficient to load them on demand.