Runescape has the system called the Combat triangle which works as the paper, scissor, rocks.
- Meele is strong attacking ranged armours
- Ranged defeats Mage's
- Magic defeats Meele
They have it easier, because they have a no class system. Which can be modified depending on what they wear, that way you are not bound to any stereotypes such as that the mage can not wear a heavy armor; a mage must look like Gandalf and due to his age he must tripp over his beard. In this type of world, players can be a hybrid of everything. That way armored mages could be effective fighters but their maneuverability could be weakened. They wouldn't be able to jump. fly, swim or do something that was possible in robes, such as creating a cyclone to life off the ground(came up with this now).
In Tibia, it has been a serious problem for Cipsoft to balance the classes. They had one class which has been a hybrid - the paladin
- Knights - Heavy armored had high defence must approach enemies in close combat has lot of HP, easy to play, has a non degradable weapon.
Their magic level raises slowly .
- Paladins - (rangers) Hybrid of a mage and knight. They use bows and other ranged equipment to fight, they have only 30% less health than knights but they have 3x more mana than knights. Their distance fighting skill raises fast and their magic level raises a lot faster than the knights. Their defence goes slower than knights, just about 30% slower, while the mages have the defence level progression slower by around 95%.
- Druids - healers that were also earth/ice Mages. In combat they were mostly healers, and they also supplied special healing runes to all other. They are very expensive to play at a higher level and require a massive ammount of money to fight monsters(they need to drink mana potions that are used for casting spells).
- Mages - had most powerful spell, and were the only class that could create an item used by both Fire/Energy Mages. They also act in combat like the druids.
Cipsoft has changed some stuff in every update he delivered. When I was playing in 2008, they have downgraded the effect of healing runes that were player made, which wrecked the player based economy, the druid class has been useless, and the knight has been fairly weaker. The paladin, considered to be a balanced character, received a special weapon called the assassin star, which made the entire class 50% more deadly. In order to make up the failure, Cipsoft has weakened the weapon and made the entire class weakened. In the same time, the druids got a pump with a stronger spell. They have been keeping this up for the last few years. Leaving out the balanced classes out of the game system.
I would suggest giving every class a special ability, not an "iron fist" but a special skill, that is needed around the world, such as some kind of elemental magic, a druid in Tibia is effective against the fire creatures, where the Mage would do little harm. For roleplay make an ability, that the stronger Tank class will be good and the hybrid of both will have both of their weaknesses but this would also make them more flexible. So the tank healer will be able to sustain a very strong attack. I do not know your system so giving you my solutions to a situation I do not know.
You should look up the Path of Exile skill tree, which is the most complex level-up system in the gaming world.