Since you have asked multiple questions at once I will try to shed some light on the whole topic.
Composing waves of different enemy types
Similar to David M's answer I would suggest assigning an cost to each enemy type reflecting it's strength or difficulty. Then for each wave you have an budget which you can spend on different enemies which make up the current wave. By increasing the budget each level you can increase the difficulty of this level.
The strategy for distributing enemy types is up to you. You can spread them out evenly so each wave is essentially the same just with more and stronger enemies.
I would instead suggest to have different wave distributions to make the waves more distinct. Each level you choose one distribution type at random which will prefer e.g. mostly ranged enemies, mostly mages and some melees to protect them or an balanced distribution among all types. This way you can give each wave an distinct character and force the player to come up with different tactics for each wave type.
This way you avoid having the same enemies in every wave which will result in the same tactic and same gameplay every time, just with more enemies.
If you have wave types with only strong enemies you should only enable them if you have reached a high enough budget so you wont have a wave with only one very strong enemy early on in the game.
You can adjust or increase the difficulty for each level/wave with several methods:
- Increase number of enemies or choose stronger enemy types (increase budget)
- Make enemies smarter or give them new attacks
- Increase per enemy strength (e.g. more health, more damage)
In my personal opinion increasing the number of enemies and switching to more stronger enemy types should be your main method to increase the difficulty of a wave. Boosting their AI or giving them new attacks, skills or abilities is very good but often difficult to archive. Increasing the per enemy strength often feels weird and results in 'bullet sponges' (enemies with seemingly endless health) or enemies which can one-hit you, both behaviors which, in my opinion, are signs of bad game design.
This heavily depends on what type of game you want to create.
If you want to have some sort of highscore or generally want to make the game competitive in some way you need a fixed difficulty. You just increase the difficulty (e.g. by increasing the budget) each level/wave by some amount and wait for the player to die after level X because he isn't able too keep up with the increasing difficulty anymore. This allows the player to feel good after beating wave X because it's an archivement requiring some level of skill. The downside it that this can result in a player failing too often and feel weak, eventually give up after dying too often.
If you want the player to feel strong but always give them a challenge without overtaxing his abilities you may want to go for a dynamic difficulty. You slowly increase the difficulty and measure the players performance each level. If he performs well you increase the difficulty, if he fails or struggles you step down a bit and give him easier enemies. This will result in an more 'fun' gameplay but now you cant really assign some sort of highscore anymore because he only beat his own personal difficulty which is not comparable to want some other player has archived.
On this last topic of adjusting the difficulty I can recommend these sources:
- Vales AI-Director from Left for Dead (starting at page 51)
- This YouTube video on dynamic difficulty in general