Your conclusion that "the heavier the better" is not wrong. You have to think of it in terms of an important detail of these game-design issues: trade-offs. It's generally important to make game choices have different costs, so the player can balance her/his choices depending on the gameplay she/he aims, knowing that each choice will have game-play cost (besides buying the item, of course).
So, suppose as you said, a 3x3 matrix that describes loosely the relationship between each weapon and each armor. The trade-off can be put either directly within the matrix, or left to an external element that is outside the matrix. Let's see.
1) Trade-off inherent to the weapon/armor relationship (within the matrix)
For example. If you think of weapon-armor relationships in which each weapon is more suited for each type of armor, your matrix is similar to the following:
Leather Mail Plate
Blunt + - -
Pierce - + -
Blade - - +
+ = strong; - = weak
In that, you can clearly see that Blunt is better for Leather, Piece is better for Mail and Blade is better for Plate. So in that example Blade against Leather is not as effective as Blunt against Leather. Here, the trade-off is given by the matrix, is inherent to the matrix: when you choose a weapon, you already know that you will be the better equipped against one type of enemy, but less equipped against another type. That's what typically is the case with fire/water specialization spells in RPG games
2 ) Trade-off by an element outside the weapon/armor relationship (outside the matrix)
However, someone could argue that in that specific example, it is not very realistic that Blade is less effective against Leather than Blunt. Blade is powerful against any of these armors. So, someone could instead think of a different type of relationship matrix for these armors and weapons:
Leather Mail Plate
Blunt + - -
Pierce + + -
Blade + + +
+ = strong; - = weak;
Notice that now, there is no trade-off within the matrix: Blade is the best choice because it's good against three types of armors. Pierce is the second best and Blunt is the worst because it's only good against Leather armor. So, to avoid Blade being the obvious choice ever, the trade-off is put outside the matrix, i.e. in an element that is not represented in the matrix. For instance, armor and weapon weight (and its consequences to mobility, agility, etc).
That's why armor/weapon powers vs. mobility/agility is indeed among the most common trade-offs. For example. The reason for the armor to follow the path "the heavier the better" is because while you get more protection from the better armor, they might make you slower, or diminish your ability to carry more items. Then, you trading strength per agility, or even defense per attack if in your game agility is important for attacking.
In another front, weapons can also follow the heavier-the-better rule for similar reasons. Heavier swords of course tend to do more damage in a medieval-ish setting. But they have to occupy both hand s of the character, or make she/he hit less times per minute, etc.
All that said for conceptual explanation as you asked, nothing prevents you from thinking of armors and weapons in different ways. If you are not happy with the second type of matrix, you can certainly aim for the first. It means, you can choose weapons and armors that are each appropriate for each case instead of relaying on external elements to create the trade-off.
So, to sum up, there is no single answer for your question. The most correct answer is: you have to choose a trade-off between what the armor or weapon gives to the player, and what it takes from the player (i.e. in terms of gameplay mechanics, besides the cost to buy). If the power vs. agility/mobility does not appeal to you, you can totally come up with another more fitting to your game depending on its characteristics.
EDIT: note that the answer was heavy edited for clarity, although the content is the same.