Flying AI shakes when chasing and avoiding obstacles simultaneously

Question

I've got a problem with my flying AI in Unity. When it just chases a target it's ok, but my system contains avoiding obstacles so when it starts to do it, it shakes weirdly.

As I understood using Debug.Log, that happens because of simultaneous switching between chasing and avoiding obstacles. It's been biting me for several days, but I still don't know how to fix it.

Here's my code:

using Cinemachine;
using System.Collections;
using UnityEngine;
using UnityEngine.UI;

public class EnemyNew : MonoBehaviour
{
    [Header("Settings")]
    public int HP;
    public float speed;
    public float detectionRadius;
    public int gunsCount;
    public Vector3 stoppingDistance;
    public LayerMask obstacleLayer;
    [HideInInspector]public bool alive;
    [HideInInspector]public float distance;
    [Header("Components")]
    public Rocket target;
    private AudioSource audioSource;
    RaycastHit hit;
    void Start()
    {
        alive = true;
        audioSource = GetComponent<AudioSource>();
    }
    void FixedUpdate()
    {
        distance = Vector3.Distance(transform.position, target.transform.position);

        if(distance <= 100)
        {
            if (target.isAlive)
            {
                Vector3 direction = target.transform.position - transform.position;
                Physics.SphereCast(transform.position, transform.localScale.magnitude, direction, out hit, detectionRadius, obstacleLayer);
                if (hit.collider != null)
                {
                    Debug.Log("Avoidence");
                    Vector3 avoidanceDirection = Vector3.Cross(hit.point, hit.normal);
                    transform.position = Vector3.MoveTowards(transform.position, avoidanceDirection, speed * Time.deltaTime);
                }
                else
                {
                    Debug.Log("Movement");
                    transform.position = Vector3.MoveTowards(transform.position, target.transform.position + stoppingDistance, speed * Time.deltaTime);
                }
            }
        }
    }
```

Answer 1

If I'm understanding correctly, the issue is that you're oscillating back and forth between two states...

State 1 is when you're out of collision range and it moves normally.

State 2 is when it's avoiding an obstacle.

Since there are only two states, you're going to hit a scenario where avoidance moves you just far enough away to switch to moving, which could move you back into range, causing you to start avoiding, etc, etc...

Instead of having a hard, binary transition between the two states, why not fade/ease between them?

For example (untested):

if (target.isAlive)
{
    Vector3 direction = target.transform.position - transform.position;
    Physics.SphereCast(transform.position, transform.localScale.magnitude, direction, out hit, detectionRadius, obstacleLayer);

    //Calculate the "normal movement" position every time
    var movementResult = Vector3.MoveTowards(transform.position, target.transform.position + stoppingDistance, speed * Time.deltaTime);
    if (hit.collider != null)
    {
        Debug.Log("Avoidence");
        Vector3 avoidanceResult = Vector3.MoveTowards(transform.position, Vector3.Cross(hit.point, hit.normal), speed * Time.deltaTime);
        var distance = (transform.position - hit.point).magnitude;
        if(distance < ???) {
            // We're really close, just avoid.
            movementResult = avoidanceResult;
        } else {
           // We're close but not too close. Blend between normal movement and avoidance.

           var invProximity = distance / detectionRadius;
           movementResult = invProximity * movementResult + (1-invProximity) * avoidanceResult;
        }
    }
    transform.position = movementResult;
}

That way, as you cross the boundary where your avoidance code is triggered, there's no significant change, but as you get closer, you start avoiding harder.

It's up to you where you set the inner and outer "boundaries", but the idea is that outside the outer boundary, it's normal, inside the inner it's avoidance, and in between you cross fade.