I assume that this API method you're talking about projects the point onto camera's near plane. That would mean that you get point on screen so that
X=-1 is left border,
X=1 is right border,
Y=-1 is bottom and
Y=1 is top. If that is the case, then you have to clip this X, Y point to your camera boundaries, which should be in normalized coordinates as [-1, 1], something like
icon.X = max(-1, min(X, 1)). That should do the trick. You do that in both dimensions X and Y.
Additional check for world point being "behind" player could be good to manually snap the icon to the bottom of the screen: if waypoint projected onto player's forward vector points back AKA
dot(forward, wapointPos-playerPos) < 0, then
icon.Y := 0.
Another way (assuming you already have a code that checks if world point is in camera's cone of vision) is comparing the sign of dot product between the waypoint vector and camera's right. If
dot(right, waypoint) > 0 then icon is on the right border, otherwise on the left. This generalizes to 3 dimensions by repeating this check for Y. You just have to think in planes
camera.forward, camera.right for positioning on left/right border and in
camera.forward, camera.up for positioning on the top/bottom border.
If you lack the code to check if point is inside camera's cone of vision, then you would have to do something like this:
- project the waypoint onto camera's
- clip the point to camera's boundaries.
You can use intercept theorem to project the point onto camera's near plane (check out the picture). Additional work is required to translate the projected vector to normalized coordinates of near plane (for example calculate using FOV and projecting on camera's right vector)
// waypoint vector
let W = (waypointPos - playerPos);
// x like in picture
let x = (near * W.length() / dot(W, forward))
// waypoint projected onto near plane (forward is camera's forward vector)
let W' = W * x / W.length() ; // watch out for overflow
// check if camera waypoint is behind neaer plane of camera
let glueBottom = dot(forward, W') < near;
// translate W' to screen coordinates [-1, 1]
Key aspect is projecting vectors onto right planes - but planes are always determined by two other vectors, so you just use them to project correctly.