For instance, in Half-Life 2 you have Civil Protection, Combine Soldier and Combine Elite. Each class has a weapon. Is there a shift away from this at all, where the NPCs are matched with the player in the sense that the weapon they choose to use is based upon their advantage, while still maintaining some sense of competitiveness? Is there a reason this became the norm? Does it have to do with all the variables and complications that can arise from doing that?
There are two reasons I can see immediately, both to do with simplicity: it's simpler to program the AI, and it's simpler for the player to understand. Now for certain games it may be an advantage if enemy characters can switch weapons for the situation (for starters, it's more realistic) but for most games it can be a significant problem if the player gets confused about what tactics to expect and counter. Players like being able to think "oh there's one of the grenade dudes, I better watch out for grenades."
As for complication in AI, there's just plain less thinking for the enemy to do if it doesn't have to consider which weapon to use. Consider your own thought process when deciding which weapon to use; now try to model that thought process as a set of instructions. You'll be considering damage per shot with how accurate the shots are, weighing effectiveness against different sorts of targets, and balancing all those factors against how much ammo you have. Pretty complicated huh?