Managing AI state transitions in Unity

Question

I am building a horror game in Unity, and I am trying to make an enemy movement controller. The way it works, is the enemy has two states, made available via an enum called EnemyStates the possible values are Chasing and Patrolling. When the enemy spawns it, it defaults to patrolling, which when done makes them follow a specific path set by waypoints on the map.

When a gameobject tagged as 'Player' is within chase distance, the enemystate is switched from 'Patrolling' to chasing, and the target is set to the player gameobjects transform. The mob then begins chasing the player.

I am doing all of my checks in the Update() function, which works mostly, but causes problems in this area;

Since it's checking every frame, it's overwritting it's previous state even if it's the same as the last time. When the enemy is patrolling then switches to 'chase' state, it plays a special one-time animation via a SetBool() with an exit time. The problem is that since it's in the Update() function, it's continually trying to call the one-time animation overwritting itself and not doing anything.

I am trying to find the best approach to handle target change etc without relying on Update() to do it, but still making the target update correctly when either 1.) A new player gets closer and the enemy starts chasing them, or 2.) No players are within range, and the enemy goes back to patrolling.

I was thinking, maybe it would be best to put the distance check on the player, and when the player is close enough to the enemy, update the enemies target and enemystate directly ? Then when the player gets out of range of the enemy, it updates the enemies target and enemystate back to patrolling?

Is that the best approach, or is there something else I should be doing? I have attached my current bit of the mobController.cs that I am working with.

using System.Collections;
using System.Collections.Generic;
using UnityEngine;
using UnityEngine.AI;
using TMPro;


// Need to redo all of this. The first thing the mob should do after it spawns, is head to the first waypoint. Then check if a player is close by, and if so set them to the target.
// We also need to build this so that if the monster is currently chasing somebody, they first do the 'Howl' animation, proceeded by the chase animation (which should be a run).
// Once the player gets out of distance from the monster, it goes back to the 'patrol' animation (which should be a walk or crawl)

public class mobController : MonoBehaviour
{

    public string mobName;
    public GameObject mobNameObject;
    public TMP_Text mobNameText;
    public float chaseDistance;
    public float deathDistance;

    private NavMeshAgent agent;
    private Animator animatorController;

    private float defaultspeed;
    private int currentWaypoint = 0;

    enum EnemyStates
    {
        Patrolling,
        Chasing
    }

    [SerializeField]
    GameObject target;

    [SerializeField]
    AudioClip chaseMusic;

    [SerializeField]
    Transform[] waypoints;

    [SerializeField]
    EnemyStates currentState;
  
    // Start is called before the first frame update
    void Start()
    {
        agent = GetComponent<NavMeshAgent>();
        target = this.gameObject;
        animatorController = GetComponent<Animator>();
        float defaultspeed = agent.speed;
    }

    // Update is called once per frame
    void Update()
    {

        // Loop through every player, see if any are close. If they are, set the mob to chasing. If not, set it to patrolling.
        GameObject[] gos = GameObject.FindGameObjectsWithTag("Player");
        GameObject closest = null;
        
        Vector3 position = transform.position;

        float localdistance = Vector3.Distance(target.transform.position, transform.position);

        if (localdistance < chaseDistance)
        {
            //Debug.Log("You are being chased!");
        }

        // Loop through all players, see if the mob is near anyone. If so, and it's not the original target, start chasing them instead.
        foreach (GameObject go in gos)
        {
            float distance = Vector3.Distance(go.transform.position, transform.position);

            // This will need updated eventually. Right now, if the player gets above the chase distance, the mob will go back to patrolling.
            // We should probably make this a decision based thing. (Should the mob follow indefinitely, until another target gets closer?

            if (distance < chaseDistance)
            {
                target = go;

                target.GetComponentInChildren<AudioSource>().clip = chaseMusic;
                if (!target.GetComponentInChildren<AudioSource>().isPlaying)
                {
                    target.GetComponentInChildren<AudioSource>().Play();
                }

                // Play the howl animation to start the chase
                if (currentState != EnemyStates.Chasing && target != this.gameObject)
                {
                    animatorController.SetBool("move", false);
                    //animatorController.SetBool("howl", true);
                    StartCoroutine(StopHowl());

                    currentState = EnemyStates.Chasing;
                    animatorController.SetBool("chase", true);

                }
            }
            else
            {
                currentState = EnemyStates.Patrolling;

                if (target != this.gameObject)
                {
                    target.GetComponentInChildren<AudioSource>().Stop();
                }
            }

            // If the monster is close to a target player, do the death sequence.
            if (distance <= deathDistance)
            {
                agent.isStopped = true;
                animatorController.SetBool("move", false);
                animatorController.SetBool("attack", true);
                target.GetComponent<CharacterController>().GameOver();

                // Then reset the target to self until next update
                target = this.gameObject;
                currentState = EnemyStates.Patrolling;
            }
        }

        // If the mob is not chasing anyone, return to waypoint patrolling
        if (Vector3.Distance(transform.position, waypoints[currentWaypoint].position) <= 0.1f)
        {
            Debug.Log("Updating waypoint");
            currentWaypoint++;

            if (currentWaypoint == waypoints.Length)
            {
                currentWaypoint = 0;
            }
        } else
        {
            Debug.Log("Too far away from waypoint");
        }


        if (currentState == EnemyStates.Chasing)
        {
            agent.destination = target.transform.position;
            //agent.speed = defaultspeed + 2;
            animatorController.SetBool("chase", true);
        } else
        {
            agent.destination = waypoints[currentWaypoint].position;
           //agent.speed = defaultspeed;
            animatorController.SetBool("chase", false);
            animatorController.SetBool("move", true);
        }
    }

    IEnumerator StopHowl()
    {
        yield return new WaitForSeconds(1);
        animatorController.SetBool("howl", false);
    }
}

Answer 1

About a third of the way into your Update() function, you have this:

 // Play the howl animation to start the chase
if (currentState != EnemyStates.Chasing && target != this.gameObject)
{
    animatorController.SetBool("move", false);
    StartCoroutine(StopHowl());
    currentState = EnemyStates.Chasing;
    animatorController.SetBool("chase", true); //<----------
}

Later in the Update() function you have this, which is called every frame as long as the enemy is in the Chasing state:

if (currentState == EnemyStates.Chasing)
{
    agent.destination = target.transform.position;
    animatorController.SetBool("chase", true); //Why are we calling this again?
}

Notice you call animatorController.SetBool("chase", true); in two places, the latter of which is called every frame. Presumably you don't need the second one at all.