Writing up an answer from my comments as I believe it's not such a crazy idea after all.
- The "Up" direction should not matter in these calculations
- You have found the object in focus and you can ray cast from its front just like you did with the character
- The "Up" direction is Z. (This could be Y depending on who you are talking to or what engine you are using)
- Front is defined as front 180 degrees of the humans view.
- Back is defined as the back 180 degrees in back of the human model.
1) cast a ray from the front of the object in focus. In the case of a human you have a definite front and back.
2) Find the diference in x and y so you can translate the ray from the object in focus on top of the players ray
3) Apply that translation. You should now have what is viewed in picture 3. This allows you to solve for the angle (theta in this case). There are details on solving for theta here: http://www.wikihow.com/Find-the-Angle-Between-Two-Vectors
Just some simple linear algebra. Nothing too bad I think.
1) Now, intuitively, you can see that if this angle theta is >90 degrees then the player is facing the "front" of the object in focus.
2) If the angle is <90 degrees then the object in focus is facing away from you, and the player is looking at its "back".
Remember the assumption mentioned before on "front" and "Back". Depending on your gameplay needs you may want to define this differently. WoW -seemed- to have this kind of setup when it came to calculating if the player was "in back" of the target. Playing as a rogue, if you are in the back 180 degrees you seemed to get the damage modifier for being "in back" of the target.
Good luck. Hope this helps.