So: top-down game, my enemies chase the player, when they get within a certain distance they stand still and fire. If they're all coming from the same direction they all end up standing in the same spot (i.e. standing "within" each other), as I'm not currently doing collision detection between enemies - they are free to pass over each other.
What's a simple way around this? Either some form of collision detection or some ai?
You don't need collision detection, but you will need to give your enemies a bit of intelligence, and have them avoid running in to each other. Collision detection without that will just make your enemies look stupid anyway - people avoid each other, they don't generally collide.
Look up "flocking" for some simple behaviours.
The basic idea is that things should head towards the player, but away from a close obstacle (each other, walls, that kind of thing).
You could inspire yourself on flocking behaviours, such as Boids.
These tend to consist on 3-4 very simple rules that, applied correctly, result on very satisfying and "complex-looking" behaviours.
In your case, you could try something like this:
What you're asking about in general is dynamic avoidance between actors. You could take a look at velocity obstacles: http://gamma.cs.unc.edu/RVO/
You may want to check out collaborative diffusion (great animations here) or some other kind of antiobject pattern for movement. While the diffusion algorithm in and of itself may not stop enemies from crossing over, you can add additional constraints to make it unlikely or impossible.
What if you disallow moving through other units? It seems to me you will need it sooner or later. Whenever the current path is blocked by a unit, adjust the path to move around that.
There's some reference for steering and flocking behaviours at Craig Reynold's website. They're relatively simple to implement but can produce some lovely emergent behaviour.
All the other answers produce a better result, but the cheapest(?) way to avoid complete grouping is to just add a random vector that is perhaps 25% of the original calculated movement vector so that objects bounce around a bit in a general blob group. It won't keep things completely off of each other, but depending on what you are trying to achieve, I have found this can be adequate. Objects can still group with this method if you run circles around them, but over time they tend to diffuse.
Here is a small game-let I made several years ago that demonstrates this method: http://deleter.phatcode.net/index.php?page=projects&p=4
People always bring up boids because that gives very good results, but implementing boids may be overkill for your specific purpose. In the past I've gotten very good results by simply randomizing the target position within a small circle.
For example, for a demo I built in Unity I used its command Random.insideUnitSphere to do flocking puffballs
I was facing the same probelm when I using NevMeshAgent as enemy, enemeis are penetrating in eachother so i increase the
Thanks to gameDevAnswer
