# Transforming world to canvas goes wrong when camera is moving

I am attempting to draw a box around a selected target in my space game. I had this working fine using normal techniques when my canvas was set to Screen Space - Overlay. However my setup is that ship models are fully present and physical and inside the cockpit of each one there is a "CameraRig" transform that by default the player camera attaches to unless they are free looking or moving around the ship.

With this setup the GUI would draw over the walls of the ship and it looked silly so I changed the canvas type to Screen Space - Camera and set a plane distance so that the gui was outside of the ship. I modified my box calculations to use RectTransformUtility to convert the existing screen position to canvas position.

This all works fine when the game is first ran and the ship is sat looking at a sun positioned at 0, 0, 0: and it renders correctly if I free look the camera around. However the moment I set a new target and the ship starts rotating and moving towards it something goes wonky and target boxes appear in strange places, often directly to the side or above/below so i have to free look around to find them and often with wrong width/height. Here is an example where the planet im moving towards is centered but the target reticule is off to the side and will ultimately not be visible unless i hard look left:

If I then fly back to the sun often the shape of the box will be wrong and slightly maligned, so I feel like this is to do with the cameras position and rotation. But I cannot control it, its hard attached to the camera and controlled by unity and the unity methods I've used to calculate the position are supposed to account for camera position/rotation, right?

My code (edited to only show relevant):

var center = targetRenderer.bounds.center;
var extents = targetRenderer.bounds.extents;

var worldCorners = new[] {
new Vector3( center.x + extents.x, center.y + extents.y, 0 ),
new Vector3( center.x + extents.x, center.y + extents.y, 0 ),
new Vector3( center.x + extents.x, center.y - extents.y, 0 ),
new Vector3( center.x + extents.x, center.y - extents.y, 0 ),
new Vector3( center.x - extents.x, center.y + extents.y, 0 ),
new Vector3( center.x - extents.x, center.y + extents.y, 0 ),
new Vector3( center.x - extents.x, center.y - extents.y, 0 ),
new Vector3( center.x - extents.x, center.y - extents.y, 0 ),
};

var screenCorners = worldCorners.Select(corner => cam.WorldToScreenPoint(corner)).ToArray();

var canvasCorners = screenCorners.Select(c =>
{
c.z = 0;
RectTransformUtility.ScreenPointToLocalPointInRectangle(canvasRect, c, WorldCanvas.worldCamera, out var transformedPoint);
return transformedPoint;
}
).ToList();

var screenCorners = worldCorners.Select(corner => cam.WorldToScreenPoint(corner)).ToArray();

var canvasCorners = screenCorners.Select(c =>
{
c.z = 0;
RectTransformUtility.ScreenPointToLocalPointInRectangle(canvasRect, c, WorldCanvas.worldCamera, out var transformedPoint);
return transformedPoint;
}
).ToList();

maxX = canvasCorners.Max(corner => corner.x);
minX = canvasCorners.Min(corner => corner.x);
maxY = canvasCorners.Max(corner => corner.y);
minY = canvasCorners.Min(corner => corner.y);

var width = maxX - minX;
var height = maxY - minY;
var posX = minX;
var posY = minY;

GuiImage.anchoredPosition = new Vector2(posX, posY);

GuiImage.SetSizeWithCurrentAnchors(RectTransform.Axis.Horizontal, width);
GuiImage.SetSizeWithCurrentAnchors(RectTransform.Axis.Vertical, height);


if you would like to see full update method unedited then here is a paste: hastebin (scaleby is 0, usedeltas is false).

The camera the player is looking through and the camera used by the canvas are the same camera.

I am at a complete loss as to why it is behaving this way. Welcome all input!

I realised an old test blanked out the Z in the worldCorners to just 0 and was never reverted for future refactors - I later fixed my bug but this remaining meant the screen position step was never completely accurate - except for an object at 0, 0, 0 - the sun!

For anyone attempting this in the future the correct steps are:

• Calculate the corners of the object in world space using their center and extents in above code (add in Z)
• Convert them to Screen Space using the WorldToScreenPoint method in your camera
• use RectTransformUtility.ScreenPointToLocalPointInRectangle to convert to the canvas space by passing in your canvas RectTransform, the position and the camera used by your canvas.

You can now use these corners to calc the x/y and width/height. When using with a Image component you should set the anchoredPosition and set the width height using SetSizeWithCurrentAnchors.

Final caveat,this expects the image anchor (position and pivot) to be set to center else everything will appear misaligned. Hope this saves someone some time.

Edit: Extra thing I encountered after all this and how to fix it for future searchers: if as you move around you find your canvas elements shaking/vibrating lots you can mitigate it by selecting your camera and changing the Near Clipping Plane value to something higher than the default 0.3 - the higher it is the less shake so the highest that works for your game.