First thing to note is that you only need to consider corners. The player will almost always see a corner of the box before an edge. In the rare cases where the alignment is just right, the player will see an edge and a corner at the same time (because every edge obviously contains corners). So it's enough to only consider corners.
The problem can be solved with linear algebra easily in 2-D, the generalisation to 3-D needs a bit more maths but nothing particularly advanced. The basic algorithm is:
1) From the position of the player and the angle of the line, create a linear equation y = m*x + c which defines the line.
2) The minimum distance between a point and a line is given by the length of a line segment perpendicular to the original line which reaches the point. So, from y = m*x + c construct a unit vector u perpendicular to it.
3) For each corner of rectangle p, find the value t such that p + tu lies on y = mx + c.
4) If all the t's are the same sign, it means that you are on the same side of all the corners. The distance to the rectangle is therefore min(abs(t)), and the corner you will see first is the one which gave that t. If some of the t's are positive and some negative, that means that you are clockwise of some corners and anticlockwise of others, and are therefore inside the rectangle.
I did the maths on the back of an envelope. The value of \$t\$ is
t = \cos(\theta) (p_y-y_0) - \sin(\theta)(p_x-x_0)
Where \$(x_0, y_0)\$ is the coordinate of the player, \$\theta\$ is the angle from the horizontal \$+x\$ axis (measured by the direction towards the \$+y\$ axis) and \$(p_x, p_y)\$ is the coordinate of a corner. This is a shockingly simple result, if I was you I would redo the maths just to double check, it's only 6 or 7 lines of algebra.
The extension in 3-D depends on how the viewer works. Is the vision of the player a wedge or a cone?