In Star Fox 64 your ship would just stop moving on the Y axis when it was at the top or bottom of the playable space. I know you said this would appear to be "jarring", but there is a way to hide it.
When your player is approaching the boundary, the skybox or whatever backdrop that lies beyond is probably significantly far away (or static) that you no longer have a reference point in view to see if you're still moving on the Y axis. As long as there are no nearby objects from the level in sight, you can maintain the illusion that the player is still flying up or down. (Make sure that cloud floor appears far away from the boundary!)
You can also reduce the FOV so that you can't see as many objects beside your character. Pull the camera back and reduce FOV (zoom in) and you will find that this trick is easier to pull off while the player is climbing or diving (looking straight up or down).
Also, if your game involves airplanes, you can constrain the min and max pitch of the plane based on the absolute Y position so that it is always flying level at the min and max Y values. Basically start lerping the (min or max) pitch between the player's input and 0° while in some region close to the Y boundary. That way there is a gradual "swoop" to level out the character before they go out of bounds, which won't appear jarring when combined with a hard limit on the Y position.