Some kind of rudimentary rules based swarming system might work. Basically, give each individual in a unit a set of rules, and treat them as individuals, not as a group. The rules though create group/formation behaviors.
For example imagine you wanted a formation of archers to stick together as best as possible, while maintaining range. And an archer can shoot a distance of 20 (of whatever unit you have).
Start with some basic rules like this...
- Archers should try to maximize their range from target (as close to 20 away, but not beyond)
- Archers should stay at least 10 units away from the target
- Archers should not have more than 2 archers between themselves and the target (to avoid overly deep formations).
- Archers should have at least 3 archers with a distance of <= 1 (to encourage grouping)
- If given an order to set up a formation at a given location, archers should set up within 5 units of the destination.
Given an order to set up formation at a certain location, the group of archers will all begin to move to the new location, evaluating their movement (or decision to not move) each time step.
- Initially, each archer will choose the target deployment spot as their destination.
- The first archer who reaches this spot will stop, and "deploy".
- Any time an archers movement destination becomes taken, the archer tries to find a suitable destination that meets rules criteria, and sets that new location as their movement destination.
- Once set up, all archers continue to check that basic setup rules are in place. If for example the enemy moves too close, archers closest to the enemy will reposition to be farther away.
It would take some tweaking, but if done right you could get some interesting behaviors as a result. Specifically, adding the "no more than 3 archers deep" rule and the "at least 3 archers next to each other" rule will drive some interesting behaviors. I honestly don't know what the result would be of those rules, but it's conceptually a way to start.
You might also have to refine this by having one archer be the leader. Let the leader make initial moves and decisions, and have the others set up based on the leader's decision. The leader might have their own rules for example to pick a new formation location when the formation gets too close to the enemy.
If you search for "rules based flocking model" you'll find a number of references to the concept that go into greater detail.