Traditional roguelikes solve the save-scumming problem by making the saving system completely automatic. The game auto-saves the player's progress whenever feasible. Especially after something went wrong (including a game over, which deletes the savegame). But the player can not create manual savegames. That means the player can quit the game whenever they want in order to progress it at a later time, but they can not restore it to an older state when they make a mistake.
This system has the advantage that you encourage the player to think carefully about their actions and experience a greater variety of gameplay situations by forcing them to accept setbacks. If done well, this can greatly increase the player's emotional investment in the gameplay.
But on the other hand, it can also be very frustrating for the player to lose a lot of progress due to a stupid mistake.
One way to avoid frustration is to make sure that starting over again doesn't feel too bad:
- Have great replay value, especially in the early game. (most roguelike achieve that with a combination of procedural level generation and a large variety of character options)
- Make sure the player feels like the game is fair. Design your procedural generation in a way that it doesn't create difficulty spikes and avoid random number rolls or hidden information which can instantly kill the player.
- Death should not feel like a complete loss. A common method is to have a progression system where achievements from one run affect subsequent runs.
Another strategy can be to avoid frustration by avoiding player setback altogether:
- Design your game in a way that the player can't die or put the game in an unwinnable state.
- Design your game so the player can easily recover from any setbacks in the game.
Regarding trial-and-error gameplay: You can have that in a roguelike, but you need to be careful with it to keep it fair. First of all, make sure that the punishment for errors is very light compared to the possible reward. Punishing the player for a mistake when they can not know what's the right thing to do can be very frustrating. Then make sure that it is possible for the player to learn something useful from experimentation. For example, let's say you have different colored potions in your game and the player is supposed to find out which color does what through experimentation. Either make sure there are lots of identical potions in each run or make the colors mean the same thing in every run. That way players have the opportunity to make use of the knowledge they acquired.