The ship stays in place and everything moves. Mostly. In this answer I present three ideas of how to go about it, and I'm referencing the games I'm basing them on.
I could not find out what approach they use for Subnautica. However, considering some of the physics glitches I've seen (note: I have not played Subnautica), I'm guessing they don't do much special about it. Except that when you are moving inside the submarine, the submarine is either not moving or moving slow enough that it usually is not a problem.
A situation where we see something similar is elevators. Traditionally games where there is a fight in an elevator will actually have the elevator static, and move the environment (for example using vertex shaders).
A relatively recent example is the game Prey (2017) that has an scripted ambush in an elevator, but it was too glitchy for combat. So for the ambush, they made an stationary replica of the elevator and teleport the player to there, hiding the teleport in the light flicker.
Quotes from "The Making of PREY - Documentary" by NoClip:
Stevan Hird: There was definitively an elevator that was a challenge. It went from being an elevator to being an elevator with combat experience in it. And trying to combine that with a lot of complex physics and some insane player power made with it.
(…)
You have a moving player on top of another moving object. And what happens when it wants to move, you know, the platforms wants to move at a certain speed, and gravity wants to shove something through it. Or it moves it from underneath you too quickly, and you don't want the player to fall. So you end up in this weird situation of trying to glue things to elevators so that physics doesn't go crazy.
(…)
Anthony Huso: The ambush, specifically, where you have that fight is not in the elevator. It's a fake second elevator. So when we do the power flickers, we're teleporting you into a fake elevator. That's where your combat takes place. And then once that thing is dead (…) we send you back to the other elevator and start it back.
Idea 1: When the player enters the ship, you teleport it to an stationary ship. Looking outside would require a camera trick: position a camera to look from the appropriate position in the world and render that. Similar to what some games do for mirrors or portals. which you could archive rendering to textures or using the stencil buffer, or clearing the depth buffer.
Some space games would usually do that for other reasons (namely to avoid floating point errors that scale in magnitude as objects move away from the origin). An example of this is Outer Wilds.
Quotes from "The Making of Outer Wilds - Documentary" by NoClip:
Logan Ver Hoef: Each planet is a rigid body, like a physically simulated object moving through space, affected by forces. And so is everything else, like everything in the game is.
(…)
Alex Beachum: The way with video games you got to keep the camera near the center of the world's coordinate space. Or you start running into floating point error and everything starts kind of jittering. And so we have to keep the player near the mathematical origin of the world. And the way we do that in this game (…) every time we apply a force to the player, we just apply an opposite force to every physics object in the world that's currently simulating. And it doesn't actually really do anything to performance because we're already doing that. We're already applying forces to every object, because everything is moving. It just another number on top of numbers. So when you jump in Outer Wilds, technically every planet's jumping out from under you, and you're more or less not moving.
Idea 2: everything moves, except whatever object the player is standing on. This means that when the player is on the ship, the ship is actually stationary, and thus moving around inside of it not a big deal.
For a more interesting example, take Portal 2. This games has an intro sequence where the room is moving. For this the player is actually moving a controller in a separate room, and then the game applies impulses to the player character to mimic what the character controller is doing, but inside the moving room.
Quote from Portal 2 in game developer commentary:
We solved the problem of trying to compute the player within this highly dynamic environment by putting them in a virtual room that has all the base shapes of the rendered container, but is simply used to compute player navigation. (It's hidden somewhere else in the map). The viewpoint of the player is then parented to the coarse simulation transform, resulting in the final rendered frame. At the end of the ride the player is teleported into the actual game space.
And yes, Sea of Thieves had to deal with that too. I believe they do physics in a replica of the ship which is stationary somewhere beneath the sea, but you see the meshes on top of the ship. At least we know they do that for navigation…
Quote from "How Megalodon, Kraken and Skeleton Ships Haunt the Sea of Thieves | AI and Games":
The big secret is the navigation mesh for the ships isn't actually moving. It's baked deep into the ocean. The navigation mesh for the entire ship is baked into the deep down on the ocean floor and each skeleton is plotting their movement on that nav mesh. This nav mesh doesn't move with the ship and stays at a fixed point. Meanwhile each coordinate on the navitation mesh is transposed from the ocean floor into the ships deck and hull respectively. This has to factor the current orientation of the ship with regards to the oceans surface. Meaning the navigation paths need to follow the actual rotation of the ship as it crashes through a wave.
Idea 3: Combining what they did for Portal 2 and Sea of Thieves we would have a physical environment that does not move where all the collisions and navigation is resolved. However, what we actually see are meshes transformed into the ship space imitating what is happening in the physical environment.
All the ideas presented here have the ship actually stationary so they don't cause trouble to the physics engine. However, I guess you want the motion of the ship to have some effect inside. In that case you might also want to change the gravity strength and direction in the physical environment, so they match the combination of the acceleration of the ship and the gravity it is under.
