Just some quick additional suggestions, that sometimes complement what others have already said.
1) water solution: I never understood why killing the player with a shark or something. Just let him/her swim infinitely (like with a proceduraly generated infinite ocean). That alone would closer resemble the idea of how distant it would be in real life to cross an ocean by swimming. The player would just get tired at some point of going forward and then would decide to reload (because going back all over again would be just as tedious).
Of course, if you have modern boats, ships, jet-skis, etc, and the concept of gas as resource, that works well too and then it becomes part of the item number 4 below, since the player can just sail until running out of gas. Then, if he/she can swim, it becomes again a problem of just letting him/her swim forever.
2) scene barrier: it does not have to be mountains. A dense forest, as someone else told here. A canyon with a beautiful view. A part of a city with too many buildings in the way and the only possible streets blocked. A huge fire destroying parts of a city, or a gas leak, or a radiation accident in part of a city that blocks the way. If the city was invaded by aliens, maybe the gigantic spaceship is itself blocking the path to other parts of the city as it crushed when arriving. Or, why not, you can have flying islands like in Zelda Skyward Sword, where the boundary is given by not-falling-to-death.
3) challenging who tries: you can always increase the challenge exponentially as far as the player goes. Imagine an open-world in the country fields. The player crosses farms in the direction of the inner-lands of your world. More and more enemies show up with no power-up, health kits or something in the way. If you make these field camps procedural and increase the challenge, the player will eventually get killed without having the feeling that he/she was cheated by an artificial border. I like this approach because it even creates in the mind of the players the impression that there might be something hidden ahead (of course, you have to take care and don't abuse on this illusion, to avoid the players becoming mad at you!).
4) lack of resources: if a player has to eat/take medicine/whatever, and/or if vehicles have gas, you don't need boundaries. Let the player and/or its vehicles die. That is particularly useful in a space game, where creating a non-artificial barrier is quite difficult. In other words, if that suits your gameplay framework, make the open world finite by not being accessible due to being impossible to accumulate resources to go beyond a give distance.
I won't keep going with a ton of suggestions based on storyline and gameplay. My reasons: first I don't know which type of game we are talking about. Second: frankly, as nice as it can be to have a story-line or gameplay related solution such as failing missions because some hostage has died when you tried to get out of the borders, I don't think that is exactly what you meant and, on the contrary, I found those to be usually harmful for immersion. But if you happen to be interested in these types too, let me know and I can drop some ideas that I have either seen or thought.
Also, as a general point, notice that more than once I touched procedural generation. Really, I think procedural generation is your friend here. You don't need to use it only for having infinite worlds. I always say we should start using it much more often to make the finite open-games more realistic and diverse. Using it for a fake endless border would be a neat use.
