Shaking UI World Space element when keeping it centered between 2 objects

Question

So I have a ship in the game, and I can click on a Sail to make a World Space UI panel appear. The panel will follow the selected sail and also always Billboard towards the camera.

The problem I have is that the UI element will start shaking after a few seconds of gameplay. World Space position of the objects is near Origin (0,0,0) so this is not a floating point issue I think?

The camera can freely rotate around the ship, so the UI element needs to keep track of the moving sail(because the ship is moving) but also keep track of the camera position.

My code:

public class UIWorldSpacePanel : MonoBehaviour
{
    Camera cam;

    public MeshFilter FollowMesh;
    public float MidDistance = .5f;
    void Start()
    {
        cam = Camera.main;
    }
    void FixedUpdate()
    {
        transform.position = Vector3.Lerp(FollowMesh.transform.position, cam.transform.position, MidDistance);

        transform.LookAt(cam.transform.position);
    }
}

I also tried this:

void FixedUpdate()
{
    Vector3 oldPos = transform.position;
    Vector3 newPos = Vector3.Lerp(FollowMesh.transform.position, cam.transform.position, MidDistance);
    transform.position = Vector3.MoveTowards(oldPos, newPos, Time.deltaTime);

    transform.LookAt(cam.transform.position);
}

But this doesn't work as the UI element just flies away into the distance, it wont do that if I turn off billboarding, but that's useless...

Putting the code in Update or FixedUpdate doesn't seem to fix the issue either.

Answer 1

Using LateUpdate instead of FixedUpdate fixed the problem for me:

void LateUpdate()
{
    //code
}