So I've been working on my game for roughly two years now and although it's significantly more balanced than it was a year ago, I still feel like it's too easy, or rather, doesn't really have any sort of difficulty curve over time. It's a top-down shooter, and more enemies appear as you get to later waves, but the bosses are all a joke in terms of difficulty and some of the weapons can easily obliterate large groups of enemies if you know how to use them properly.
So how do I know what "magic numbers" to pick to have a consistent difficulty to my game, and is there a good way to determine a formula to increase that difficulty in later waves? Should I be increasing health of enemies over time? Or would that not make sense since they're the same enemies? I increase number of enemies over time by having a "spawning pool" which is just an integer that increases on a logarithmic scale according to a formula I just came up with and plotted a while ago. Each enemy has a "cost" to spawn and at the beginning of each wave, random enemies are chosen and their cost deducted until the spawning pool is depleted for that wave.
The enemies' damage and health are calculated randomly using a sort of D20 system like Dungeons & Dragons where, for example, the very first enemy you encounter in the game's health is calculated by rolling 2d8+12 (roll two 8-sided die and add 12), so the range of health values they can have is 14-28. All enemies' health is calculated this way. The enemies' attacks and damage over time effects, as well as the player's weapons all use this method to generate numbers.
Is there a good way to decide on good numbers short of just playing over and over again until it feels right? Would it be bad to increase their health and damage over time since they're still the same enemies? Or maybe I should get rid of them in later waves and replace them with an upgraded basic enemy that's identical (maybe a color swap) but with better stats?
Examples would be awesome.