Way over 90% of the performance cost seems to be coming from colliders parented to animated bones. If I disable Motion & Animation I can spawn 20 soldiers with around 1ms cost on my mid-low end system. Unfortunately even when the soldiers are "not moving" the idle animation requires the colliders to move with the bones. With motion enabled the performance of 20 soldiers is roughly 15ms. I managed to improve performance by creating a simple hitbox (see left photo), the performance of 20 such soldiers is 10ms. Removing all colliders from the Soldier again lowered performance cost down to 1ms.
The hitbox colliders have been disabled for all collisions in the Physics Manager. Obviously they are not set as static. They are for raycast detection ONLY.
I have developed a 6th sense for when I'm "doing something wrong". Are moving colliders really that expensive?
Script attached to Collider: