A while back I sat down and tried to come up with all of the fundamentally-different MMO combat roles I could, and I ended up with the following list. Note that this is pasted pretty much directly out of my voluminous ideas file and makes a lot of references to WoW, since that's what I was playing when I wrote this up.
Standard trio
DPS: Kill monsters through sheer quantity of outgoing damage.
Tank: A tank's sole job is to take a lot of damage (or, rather, to not take a lot of damage when anyone else would be taking a lot of damage.)
Heal: Repair damage done to tank or to other players.
Conventional support
Buff: Make the group more powerful. Difficult to make fun - you'd need a situation where the optimal buffs were constantly changing.
Debuff: Make the enemy monsters less powerful. Again, difficult to make fun.
Crowd control: Completely nullify single enemies or groups of enemies so that the players can focus fire more effectively.
Screening/defense: I'm thinking there's a distinction between "heal damage that has been taken", "be a tank", and "prevent damage to other players". Some kind of a super-discipline priest. Fundamentally different, but difficult to show that.
Unconventional support
Sniper: Class whose sole purpose is to nullify single important parts of enemies. "Spell interrupter" would be a form of this. As a slightly more complex example I'm imagining a group of enemies that have a "damage-dealing module" and a "damage-reflecting module", where only one can be disabled at a time - disable "damage-dealing" for the group that the players aren't killing, and "damage-reflecting" for the single creature the players are killing. Things can get arbitrarily complex from here.
Threat management: Usually rolled into tank, but why? Make a class that can funnel threat from one character to another, or modify how threat is accumulated, or masquerade as one character. Misdirection+Intervene+Vigilance+more.
Command: Raid leading is tough. Make a "class" whose sole purpose is to direct players, give them a bird's-eye view of the battle, stop forcing them to run their own character as well as the raid.
Mobility/positioning: We've got a little of this with Death Grip, and now Life Grip in Cataclysm, but a class whose sole purpose is to move players or monsters around for optimal placement. Imagine some kind of crazy Thaddius/Hodir combo, where positioning is vital and everyone is moving around constantly, with this dude fixing issues as they come up.
Recon: A class built around dealing with monsters that have not attacked yet - the ability to interrupt/redirect patrols, see who's coming before they come, split up groups temporarily. Rogue has a little of this with Distract.
Obviously it's difficult to come up with a way that many of these can actually be, you know, interesting, but I don't think it's impossible. You would, however, need a thoroughly different model from WoW's. Also I kind of dread the idea of "okay it's time to go run Deadmines, we need twelve players from twelve completely different classes", so I feel like any functional game would either just pick a different set/layout of these from WoW or wrap up a bunch of them into one class.
Just as an example of a possible combo, I really like the idea of Tank+Debuff+Sniper being one class, Threat+Mobility being another, and DPS+Healing being the third more common class. Your tank's job is to stay alive and keep the monsters from obliterating the party, you have one utility character constantly trying to redirect every monster to the tank and keep people out of fire, and you have three damage-dealers punching enemies and making health fly out. I think there's potential here. You can easily generate other interesting combos by just mixing and matching various roles.
It's also worth pointing out that, with these roles in mind, WoW's "standard trinity" is actually much more complicated than you might think. It's not just "Tank/Heal/DPS", it's Tank+Threat / Heal / DPS, but virtually everyone is also Buff/Debuff, DPS often has CC, Priests do Screening, and half a dozen classes have Sniper (for spell interrupts). But those are always in a rather minimal form, it's never the focus of the class.
