"Artificial intelligence" in games
In general, enemy agents in video games do not use machine learning. Instead they are expert systems, behavior tree, cellular automata, finite state machines, path-finding, and similar solutions. All of which were at some point considered artificial/computational intelligence.
These terms refer to always changing sets of technologies. For example, at some point path-finding was considered "Artificial intelligence". Currently when people say "Artificial intelligence" they are thinking of machine learning.
The rest of this answer is about machine learning and video games. Taking for granted that we can use machine learning models to assist in asset creation.
Making enemy agents with machine learning is not a great idea
A lot of the applications of machine learning to video games are because video games pose a complex controllable challenge for the artificial agents. So, it is the video game industry being useful for machine learning research, and not the other way around.
As DMGregory explains making opponent artificial agents fit into the desing of the game is a remarkable challenge, because we want them to "lose, entertainingly", and it would not be straight forward to train a model for that as opposed to training it to win. Also predictible enemies are often better as the players can steadily figure out how to defeat them.
Furthermore, machine learning solutions create friction to change (changes will often require to retrain the machine learning models) and are hard to debug (Explainable Machine Learning is an active area of research).
Examples of machine learning in video games
Again, taking for granted that we can use machine learning models to assist in asset creation.
A notable example of using machine learning in video games is augmented reality, which builds on top of computer vision. For example facial tracking and skeletal tracking implemented with pre-trained models.
We can find an examples of pre-trained models in more traditional games. For example, Supreme Commander 2 uses an artificial neural network pre-trained model to allow units to take decisions.
A more recent application is Deep Learning Super Sampling (DLSS). Yet, I'm betting that is not what you had in mind.
Furthermore, there are already experiments of using large language models and other generative models - running on a remote server - in games. But - at the time of writing - these are indie games that use it as gimmick.
Feasibility of training machine learning models during game play
So far we have talked about pre-trained models. That is because:
- Time: it is unlikely that your game session is not going to last long enough to train a model. Much less if you want to do learning at the character level, since your would be interacting with another character for just a small fraction of the game session (unless it is a character companion). And if it is an enemy character, chances are you kill it before you could tell if it was smart or not.
- Performance: You would be taking a fraction of the CPU time budget to train the AI. Remember that in video games the game already has a lot of things to do each frame to keep the game running smoothly.
Are there games that train a neural network during game play? Yes. In indie games where that is the point (games about creature evolution), or games created as part of research projects.
Addendum: There is a note worthy example: Black & White by Lionhead studios. It trained a neural network during game play, which was used as part of the pet creature behavior logic. The player could reward or punish the creature at any moment which would be applied to recent behaviors. Also, while leashed to the player the creature had a chance to mimic player actions. So positive training was the player would leash the creature, do the actions them wanted it to do, and then reward the creature for it. Negative was the player catching the creature doing something they didn't want to do and punishing it. I also want to point out that it was a very small model, which allowed for training it this way.
In fact, the situation might improve: The paper "The Forward-Forward Algorithm: Some Preliminary Investigations" released in December of last year introduces an alternative to back-propagation which is less computationally intensive and has faster training rounds (although it will require more training round to archive the same results) which might allow to train models during game play.
This does not change the fact that making opponent artificial agents work within the game design is a remarkable challenge. Yet, it makes other solutions more feasible, such as artificial agent companions that assist the player, and machine learning solutions for dynamic difficulty.