I'm working on trying to improve the pathfinding for my game's enemies. Right now, they basically just constantly move towards the player's exact position by calculating the angle between themselves and the players and moving in that direction. I also have a flocking algorithm which prevents the enemies from stacking on top of each other, so they will form up into groups rather than clip through each other.
However, now that I've added a tile-based map, I need the enemies to also be able to path around obstacles and walls for example. I initially tried adding a separation value to "non-walkable" tiles so that the flocking algorithm would consider the walls and obstacles as objects to move away from. I have yet to work out whether or not this is feasible because my initial test showed the enemies hitting an invisible "wall" where there are no non-walkable tiles, yet for some reason, they hit it and start spazzing out.
I was wondering if it might be too performance heavy to calculate a path to the player using A* and then use the flocking algorithm to prevent clumping. Originally my game was going to be a wave-based shooter, but I've decided instead to make it level-based in the vein of Hotline Miami, so it's likely I'll have fewer enemies, with the occasional horde, and just make them stronger.
Is this a viable solution? I'm using Java with Slick2D as my game engine. Or is there a better solution / algorithm that tackles both these problems?