It depends on the type of game you're playing, and I'll try to avoid overlap with others here:
Player Skill Separation
In Magic: The Gathering there are cards sometimes nicknamed "noob
traps"; these are usually Life-gain cards. While affordable in
mana-cost, these are card-disadvantage cards that offer no trajectory
towards winning. In drafting and sealed especially, these will cause
players with a low skill level and understanding to make their deck
worse inherently by choosing low value cards.
Chance Based Acquisition
In Magic: The Gathering, Pokemon, and many other games; you don't have the same chance to acquire items/creatures all the time. It's much easier to get a Hypno than an Alakazam so Alakazam is better (in ways that tend to matter.) In Magic, rare cards tend to be both more complex and more powerful, in part because during Drafts you want some "bomby" cards.
You might be thinking "that's dumb"; but people need variance in order to justify losing. It's a highly important pair of things required for a successful game:
A) That you have a chance of winning. This requires variance so that even a skilled player may lose to an unskilled player sometimes. It's even inherent in rats (study found: ncbi.nlm.nih.gov/pubmed/7250521 thanks to Theraot) that they'll stop wrestling if they don't win 30% of the time or so.
B) Additionally, humans are more likely to keep playing if they can blame the loss on luck. "Aw man, if only I'd drawn that [blorp]", "You only won because you just drew [sclub]", "I only lost because the packs I opened were garbage"
You see, this mentality, regardless of it's occasional accuracy, means you're willing to play again because it wasn't your skill that caused you to lose, it was just that you rolled too many ones (literally or figuratively.) Notice the popularity of PUBG; it exhibits both Skill Separation in the sense that you learn what to use and what to throw away, chance based acquisition, and of course is addictive (in part) because "if I had just found a [blipt] I would've been fine. Stupid game."
- Specialization
Often times people will claim that something is inferior but not realize that it does have niche use or a specific strategy that can be employed with it. In 40k, Lootas and Tankbustas (in the 4th ed Ork codex) were an example of this. Lootas were good against most vehicles and had no real drawbacks. Tankbustas were good against all vehicles, but had less average damage against the less-armored ones, and had a significant drawback. For people like myself, I found this drawback worth mitigating because their power-level against many targets was several times better. When those targets are what you're worried about it's worth paying the penalty. To most people, however, it was clear that Lootas were the best, end of story.
EDIT:
- Financial Incentive
Financial incentive is also a cause at times. In the 40k example when the Carnifex was initially released it was very under-priced and the army allowed 6 of them. This newer model and it's ability to allow (essentially) 6 heavy support slots created an entire trend for the army called NidZilla; where 6 Carnifex, 2 Hive Tyrants, and then a few troops choices for scoring or walling off opponents were the trend.
The next codex they nerfed Carnifex to nearly double the points (from mid 80s to 160-200), nerfed Hive Tyrants a bit, then a Trygon was added with double the Hit Points (around 200 pts to purchase) and a higher power level in general (attacks, initiative, deep striking, etc) at the same time they created the Tervigon as a troops choice in a similar price range as the Fex but the Terv also created dudes.
Suddenly you had scoring Monstrous Creatures that created more scoring units, and you had better Heavy Support choices; both competing for points and slots with the Fex. This is hardly the only time this has happened in 40k; and hardly the only race. It happened with Flyers in general, Space Wolves, Blood Angels, Tau (multiple times), Necrons (multiple times), a general cheapening of units to include more models in your army, and a price hike on anything that was a common choice even when it was a balanced option (Sluggas and Shootas were both viable in different scenarios, Shootas have been nerfed in both points and effectiveness.)
While I can't claim for sure that this was a marketing ploy, it certainly looks like exactly what you'd do if it was a way to sell the newer models and obsolete the old ones. Regardless of it being the case for 40k, it's going to be a case for some games.
- Purposeful Meta Game
Again, for Mtg as an example, units and sets can be released with a purposeful expectation of what a meta-game will look like; and it may be that these units are just there as clutter for you to dig through in order to find the optimal choices. They are still choices and may serve some other metagame purpose (a unit that's good against a set of units that are not useful for the intended metagame, intended to snuff out attempts to curb the metagame into something less desirable.)
See Innistrad. They wanted a metagame of Tokens, token-buffers, and some grave-strategies for longevity; but in case it was too much they included Rest in Peace, Deathrite Shaman, Grafdigger's Cage, Dryad Militant, and other grave-hosers in standard (over time) in order to be sure that the non-grave decks stood a chance. Snapcaster was a high enough power level to keep some level of grave recursion on the map and is one of the obviously best cards ever printed. Note that this ties in with Financial Incentive above.
- Legacy Units and Power Creep
By power-creeping here and there, things are "exciting" for current users. If everything new is average or below average, or at best a lateral upgrade, users won't be incentivized to purchase it or expand their collection. But if it is grossly powerful (or at least looks grossly powerful) people will purchase a lot of product to get their chase rares, to capitalize on the secondary market, or to be sure they're able to compete in the new metagame.
Even in RTS, they don't tend to add bad units; but rather things that you say "holy crap, I can detect stealth units reliably now?", "I now have a mobile unit that can harass their economy?" etc. It has to be "unbalanced" in some aspect in order for it to be interesting; and hopefully you just balance it in such a way that your old units aren't completely obsolete. See Reapers for Nod in Tiberian Sun. Reapers were a strict upgrade to basically everything. A self-healing, anti-air, good against all targets unit that could create a ball-of-death much easier than before. That's exciting.
Stealth tanks, tick tanks, etc were still useful; but the feeling of having a self-sufficient unit was big; and it was clearly unbalanced in the roles it was made for (end-game ball-of-death stuff.) No more did you use Tick Tanks for that.