Basically you're looking at modeling 6 things for a ship: pitch, yaw, roll, heave, sway and surge.
Pitch, yaw and roll are rotations the ship can make as it twists and turns going up and down the slope of the waves. Heave, sway and surge are movements induced by the waves pushing the ship around and/or the ship sliding down the face of a wave.
"Like a Car Driving on Hills..."
Imagine a boat on the water like a car driving over hilly ground. If the car drives over rolling hills (like a ship going over waves) it is going to tilt and angle as it goes up and down the hills. This is the pitch, yaw and roll. If the hills (waves) are big, the car (ship) will drive up and down, pitching, yawing and rolling as it goes. If the hills (waves) are really small (smaller than the car/ship), then the car (ship) is just going to drive over them and not pitch, yaw or roll much.
A big ship can just plow through smaller waves, whereas a small ship will move up and down the waves. Taking our car example, imagine someone riding a bicycle (small ship) over a set of small hills (waves). They will roll up and down as they go. Then someone drives a big truck (ship) over them. The truck is bigger than the hills, so doesn't really pitch up and down as it goes over them.
Unlike the car though, a ship is part way into the water, so it's movements are going to be dampened somewhat. Imagine a car with really soft spongey tires. When it drives over tiny hills, the spongey tires just smooth it out. A ship's movements are also dampened, so little waves won't make it bounce up and down like a car on a rocky road. A submarine is sort of the ultimate dampened ship, as when submerged it is pretty much immune to the surface waves. But if it's on the surface it's going to be moved by the waves.
A ship will also slide on waves. A ship going down the face of a wave will surge forward for example. So to extend our car example, make it a car with big spongey wheels driving on a somewhat slippery surface. Unless the car is running the engine to compensate for the slip, it's going to slide down the side of a hill. Even if it is running the engine, there is going to be some slip.
The one place where the car and hill analogy has problems is the fact that the waves change shape over time. A stationary ship will bob up and down as the waves go up and down.
Waves Moving the Ship
If there is no wind blowing on the ship to move it along, and the waves are a perfect sine wave shape, then the ship basically won't move anywhere as it bobs in the waves. It slides one way as it goes up the face of a wave, then slides back the other way as it goes down the back face of a wave.
However if the waves are NOT symetrical (like the picture below), then the waves are going to move the ship. Because one side of the wave is steep, the ship is going to slide quickly down that face as well as be pushed by the face of the wave. The gentle back slope of the wave however won't have much motion.
This isn't the most perfect model of wave motion and shape affecting a ship's motion, but it will probably do for a rough simulation.
Wind is also going to push your ship around in ways independent of the wave motion or ship motion. The direction and force of the wind can be different than the direction and force of the waves.
Buoyancy is how well your ship floats. Very buoyant ships float up high in the water, and ones that aren't buoyant sink. Neutrally buoyant ships (submarines) can basically "hover" at any point underwater, neither sinking nor rising. If you want to simulate a ship sinking, make it become negatively buoyant and it will begin to sink.
Buoyancy also affects dampening of the ship's motion. A ship that is extremely buoyant will bob around on the water surface and be strongly affected by waves. A ship that is less buoyant will be partially submerged and not be affected as much. Think of the difference between a pingpong ball floating on the surface versus an apple, which floats but is partially underwater. The pingpong ball bobs up and down with every wave motion. The apple on the other hand doesn't respond to every wave detail.
If the pitch, yaw and/or roll exceed some value, your ship is going to tip over. When it tips over, it might fill with water, reducing the buoyancy, thus making it not float any more.
Getting Sea Sick :o~
A ship that is traveling parallel to the direction of wave motion is "in the trough", and will produce the most nauseating effects at least in my experience :) If you are traveling in the direction the waves are going, you can have a very smooth ride - like having the wind at your back. If you are traveling in the opposite direction as the waves, you will have a pretty harsh ride as you are hitting each wave "hill" as it comes at you. Makes for a pretty exciting ride though!
Here are three articles that cover the science behind this, which might give you some insights. Though heavy on the math and science, they could give you an idea of what the different factors are.
Article 1: Modeling of Ship Roll Dynamics and Its Coupling with Heave and Pitch
Article 2: Modelling and Simulation of Marine Surface Vessel Dynamics
Article 3: Modelling and Simulation of Marine Surface Vessel Dynamics
The Author Doing Field Research
Here's me about
15 20 21 years ago when I worked on research ships :)