I am planning a game that has a level that is a confusing and illogical maze, and I want to make the player feel like giving up, but not so intense that they want to quit the game. Also, I'm probably going to have some way to restart the level if they get too lost. Do you have any ideas of subtle level design that could discourage the player from continuing?
Minor spoilers follow for Journey and Nier Automata.
Both of these games include scenes where the player is meant to feel extremely discouraged. The way they both sell this in the game mechanics is by taking abilities away from the player and making it harder and harder to progress. The important trick they use is they don't apply this adversity all at once: they do it a little at a time, gradually, over the course of a whole level/zone.
In Journey, as you climb farther up the frozen mountain...
- your "voice" range shrinks, first a little, then a lot
- your character hunches down and moves slower through the heavy snow
- the music gets tense
- you have to get closer to your partner to recharge each other
- there are fewer recharge stations, so you can barely jump/fly
- frost starts to cover you, only thawing if you're touching your partner
- strong winds push you back
- the music gets foreboding
- your flight charge begins to deplete all on its own, even when you're not using it
- enemies tear your scarf (charge meter), decreasing your maximum charge
- your scarf starts to erode and wither all on its own
- the colour drains from the scene until it's nearly monochrome, in a depressing blue-grey palette
- your movement speed slows down even more
- the music gets shrill
- blowing snow makes it harder to see where you're going
- even touching your partner doesn't thaw you anymore
- you face an empty expanse with no landmarks, so you can't even tell if you're making progress and there's no respite in sight
- the music drops away entirely, sound is muffled
- your camera wobbles and your character falters, you can no longer steer them well
- your movement speed slows down so much that you're barely moving at all
The point is that these penalties don't arrive all at once, a cue players might take as "oh, this is the hard mode area, I just have to work within these constraints". Instead, every time the player begins to acclimate to their diminished abilities or harsher challenges, things get a little bit worse, and you never feel you've gotten a handle on it.
This exploits a bias in our subjective experience of reward/punishment:
Larger rewards/punishments feel more impactful, but with diminishing returns: a penalty twice as large doesn't feel twice as bad. Most of the perceived pain/adversity comes from "I lost something at all", and the magnitude of that loss is secondary.
So, splitting a large penalty into a number of smaller setbacks spaced out in time lets the player feel each one individually. By repeatedly hitting the player with small losses, we keep them in the steep portion of the graph where they feel the adversity most intensely, even while the total size of all punishments added together is not that big (so the game doesn't become unplayable).
To use this in your game: look for things you can do that feel bad, then do the tiniest amount of them that's perceptible. Then keep doing it, over and over, with enough time in between for the next setback to be perceived as its own separate loss.
Be careful of overshooting here. As you can see in the graph, the "loss" side is a lot steeper than the "gain" side, a bias called "loss aversion" - we tend to feel the psychological pain of a loss or setback more keenly than the pleasure of a similarly-sized reward. So you probably don't have to punish as much as you think to create the desired feeling.
The player's time is often a good target: you'll notice a lot of the Journey examples reduce movement speed or range, or limiting access to faster modes the player had gotten accustomed to. Giving the character an occasional stumble, or trap they have to take a moment to pull themselves out of, all increase the sense of friction in doing anything, without killing off the player character and making them restart.
Even cosmetic changes, like altering the colour grading or vignette, or changing the character's animation to look uncomfortable or pained, can generate a powerful feeling from the player in sympathy. Nier Automata makes great use of purely visual changes to the user interface as the virus infects your character, and the display gets increasingly glitchy and information widgets become unreliable/unavailable. Sound and music changes can also powerfully sell a discouraging mood.
Something else these cosmetic effects do is help convey that there's an intention behind the player's mounting feeling of hopelessness here: it's not just that the game is buggy/unfair, it's trying to tell you a story about hardship. That clue hints to players that they should tough it out to see where the story goes, rather than just quit because the game got too hard. The mounting signs of adversity fill in for the sense of progress the player is being denied in the level design, telling them they're still getting somewhere, even if it's not closer to their goal.
Without knowing much about your game, I can't say much about what specific changes would be appropriate. But in terms of level design, some options are:
more dead ends (you can script a level where the way forward is always the last corridor they try, no matter which order they try them in)
loops, so the player feels they're going in circles (Labyrinth Zone in the original Sonic the Hedgehog has a nasty version of this)
lack of landmarks, sowing doubts about whether they're making progress
lack of guidance, so the player's less sure which way to go or whether they're getting closer or farther away
(You can make both these last two more noticeable by being generous with landmarking and signifiers of goal / proximity in earlier levels, so players get accustomed to it and miss it when it's gone)
unreliable clues (say the player has a Geiger counter whose frequency of chirping indicates when they're getting closer - start randomizing it, so the player finds they can't rely on it anymore)
terrain that's slow to cross, like waist-deep muck, or big stone blocks that need to be pushed out of the way - this is a good place to gradually crank up the slowness factor the deeper they get
Avoiding overshooting with this, where the player gets so frustrated they just quit, will be an exercise in playtesting.
I'd like to add Celeste to this as an example.
Some of the levels there can be brutal, but there is little to no death penalty and you're back where you were a moment ago. This way, you'll have to progress trough trial and error, and learning the mechanics on the way. In the extra levels, you may have to repeat a section so many times that it almost becomes a muscle memory. It can be hard, and in some times also frustrating, but it makes the satisfaction of reaching even just a little further already worth the climb.
The story also adds up to this, where it is about overcoming fears and obstacles.
The givaway here is that it is about dying often, but compensated with a quick respawn in each room.
On the other side: dying less often, but losing more progress is also frustrating, but I think that leads quicker to quitting the game.
This is a common feature of boss levels, but doesn’t necessarily have to be exclusive to them. Make the level ridiculously impossible with quick death for the player all but assured, with the ability for the player to try again over and over again to the point the player starts to think “this can’t be right; I have to be missing something”. And then they notice the key to overcoming the level (or at least surviving) that had been present through environmental cues but that they’d missed because the cues were subtle. Pleased at how smart they are and feeling accomplished the player continues the level. The level could still be difficult, even very difficult, but at least now the player knows it’s possible.