I've played Pokemon competitively in the past, and if your program is very much so inspired by Pokemon, an automatic system will be difficult.
With Pokemon, there are a few things to consider. Damage, Status Effects, Resistance, and just honest player skill. Testing for damage will work fine for Pokemon like Gallade, but not at all for Pokemon like Butterfree. Butterfree doesn't do alot of damage, but the status effects still make it a top choice.
I think the best way to implement a system would be to first rank your players. As Byte56 suggested, an Elo system will work. Then from your top 10% players (10% is just an arbitrary number), look at the frequency of Pokemon used. If you see that Mewtwo is being used in 90% of your top 10% player, you can guess there is a balance issue. From there I would do a break down of which moves tend to be most popular. In competitive Video Game Pokemon, there are tendencies for the top players to use specific builds. If you see that 90% your 10% top players use Mewtwo, and 80% of them use Psychic, then you can assume that Psychic is overpowered. So an automatic damage reduction might help balance.
But then there is the caveat of Psychic can also cause confusion in 10% of it's uses, so maybe a lowering of Psychic's confusion chance might be in order. This will just depend entirely on how complicated you want to make your game, because honestly, Pokemon is a seriously deep game.
With your level slider, I believe your system should work with rating the players, but always remember the case of the Lvl 1 Rattata that defeated the Lvl 100 Dragonite.
;tldr - Rank your players. Look at the top performers for trends. Adjust values accordingly.