In my opinion, the primary reason (apart from technical reasons) to consider having a delay before loading or restarting is:
To give dying some impact
A short delay
... is good if you're playing some simple game where you typically die in a minute. Death will rarely have an impact - you die, you restart, a minute later you're back where you were. That's really not a big deal.
You should, in general, avoid delays (intentional or not) with such games.
Death will, however, have an impact when you've reached some epic high score, or barely missed it. In such cases it could make sense to add a small delay to give the player some time to process it. This could also help with retention, in that they're presumably more likely to stop playing if you give them some time to think between games, and stopping on a high will leave a better impression of the game. One way this is (unintentionally?) done is by asking whether players want to post their score to social media.
A long delay
... can be useful for a game that tries to be more immersive.
A game with no suspense is generally not immersive - it's hard to care if you don't have some "what will happen" / "will we win" moments.
You can potentially do this with good story-telling, or having some other consequences, but the easier way and more common way to do it is with the threat of death.
The problem is that death typically doesn't really mean anything in a game - you just reload and keep going. Thus there needs to be some consequence to dying. Some games have perma-death, other take away things you earned. If you have neither of those things, you need to have some way to make the player believe that them dying is something they should actually try to avoid, otherwise they'll just stop caring when they die, and stop fearing the threat of death, which could remove all suspense from a game.
Thus you add a delay between dying and the player getting back to where they were.
In-game versus out-of-game delays
You have out-of-game delays (e.g. loading times) and, in the case of loadable games, in-game delays (how long it takes to play the game until back to where you were).
These two are usually (or should be) closely linked. If you separate those two, that can lead to problems in itself:
A short time to restart with a long time to get back to where you were in-game means players won't really be given time to process their death and decide if they want to continue, and may be frustrated by the long trek to where they died if they didn't really want to keep playing.
A long time to restart with a short time to get back to where you were in-game means players will want to restart often, so they'll be frustrated by the long waiting time.
To make players think
... could be another purpose for a delay, but this applies to different types of games (generally).
Consider some story-focused decision-based game.
Such games are meant to be highly immersive, and death may not really come into play there.
Yet you still wouldn't want players to just instantly restart or load.
You want to give them some time to process and think about what just happened.