Approach the player ship
With the major problem being for the AI to calculate perfectly the
curve required to come up along side the player, or to better yet,
make a pass along their bow / stern before coming along side without
needing to let them quickly speed up / slow down to achieve this.
You'll need to solve these problems iteratively, as there's no "perfect curve" because ships are in constant motion, with their motion affected by player twitches, sea and wind conditions. So the best your AI can do is to try to approach, updating the course every few frames (10x a second, for instance). Also, it doesn't need a full battle plan at each moment, it just needs to worry about what it needs to do to get to the next stage of the plan - "let's cross that bridge when we get to it."
I would suggest approaching on a line that is tangent to a circle of some radius representing the distance we want to be from the enemy ship. So say your maximum firing range is 500 yards, but you want to be quite accurate, so you decide on getting quite close, 250 yards. Then you aim for the nearest edge of a circle of 250 yards radius, that surrounds the target ship. Or maybe not the nearest, but rather the one that will bring you parallel to their current heading.
Keep guns trained on player
This will come second to just getting close to (or getting away from) the target. Once that has been achieved, you can begin to think about gradually changing the tactic to "line up a broadside".
Remember also that historically, the guns didn't move a whole lot inside their gunports, so you should expect maybe 10 degrees of movement to be a lot, and that's you taking artistic license! If I remember my naval history correctly, carronades were something of an exception and could be wheeled around freely above decks, but had a shorter range than main cannon.
Bring player in line
There is some variability on defining this. If you want to deliver a full broadside, you need to check that your current motion vector matches the enemy ship's, and that the centres of each ship line up on a perpendicular between those two parallel motion vectors. Otherwise you can be as parallel as you like, but you'll still miss!
Also, how parallel the two facing vectors are, is up to your own code; use the dot product here to check similarity, and some epsilon value to check equality within some margin of error. In history, there were times where they took shots even when not parallel, for example targeting the quarterdeck from astern. May even be that it depends on how desperate the AI is feeling, or how much it thinks it can gain by firing at the stern or bow of an enemy. P.S. I would imagine there is a much higher chance of deflection when firing directly at the bow, due to its shape.
Be able to manoeuvre around obstacles
This is the least of your worries, and in any case, as in many historical naval battles, collisions will happen and ships will sink if only because of the fickle nature of the ocean and winds, in spite of the crew's best efforts. Of course, any time you get too close to another ship, especially when facing vectors are different (i.e. one is going to plunge right into another), emergency manouevres will need to engage. Same applies for land or rocks.
Get the rest right first - movement, alignment, firing - then worry about this at the end.
Considering all the above...
You need to order your implementation according to importance, start simple and see what evolves:
- Proximity to enemy (approach or escape)
- Alignment (once fairly close in, but not too close, as we need space/time to do get aligned)
- Side-to-side (matching position / speed once aligned)
- Firing or boarding (once close up)
- Obstacle avoidance
In terms of the first two points, I would set up debug circles and projected curves and watch them change size / shape as AI thinking adapts to the current situation. This should really help you to visualise the situation and work with it more easily than trying to sum everything up in your head.
About realistic winds... I would not worry about that early on, if at all. For now just assume that ships are under their own power. Then if you want to, you can gradually introduce the effects of (and indeed, need for) wind-powered sailing, later.
To sum up, AI will constantly change tack (arr! maritime jokes!), and the rest relies on vector maths.