Why has this happened?
This HASN'T happened.
Duke Nukem 3D and Unreal Tournament (to use your examples) were creative games, both were trying lots of exciting new things, in new themes, in new ways.
But most other games from those eras weren't doing that.
Quake 1, for example, had a pistol and a shotgun and a machine gun and a rocket launcher and etc. So did Doom. So did Quake 2. So did Sin. So did Half Life. So did Wolf3D. So did Marathon. So did System Shock. So did Dark Forces.
Yes, a few old FPS games were inventive in their creation of bizarre new types of weaponry, but they were far and away the exception: the vast majority of older FPS games were pedestrian in their weaponry, using standard weapon tropes which roughly evoked real-world weaponry.
In the same way, the vast majority of modern FPS games are using these familiar weapon tropes, but there are a few creative and inventive modern games experimenting with new ideas (for example, Painkiller or Bulletstorm), exactly the same way that Unreal Tournament and Duke Nukem 3D experimented way back then.
There are good reasons to use common weapon tropes, instead of inventing new weapon types. They're familiar to players. They're easy to design. Players don't have to be taught how or when to use them; players already know them from all the other games they've played, or from a general familiarity with pop culture. I think I'd be shocked if they weren't the most common weapon designs across all eras of video gaming.
Which isn't to say that new weapon types aren't fun. They're just a little more challenging and time-consuming to get right, and require more explanation to teach players how and when to best use them. And if that's not what your game is fundamentally about, then I can understand why somebody might want to focus their design efforts somewhere a little more interesting than on which bullet blows up when and how big the resulting explosion is to be.