Why is reinforcement learning not widely adopted as an AI tool for agents in well-known games?

Question

Reinforcement learning has shown remarkable success in game-playing agents, as seen in the achievements of AlphaGo and OpenAI's Dota 2 AI. There are a few tools, such as Unity ML Agents, to integrate state-of-the-art algorithms into game engines, and we have recently seen that Grand Turismo 7 added an AI racer trained with reinforcement learning into their game. However, this approach to creating intelligent agents seems not widely adopted in many other complex games where AI agents such as NPCs and other opponents could benefit from its capabilities. Examples of successful integrations are still minimal.

I'm curious to know the reasons behind this trend. Why is reinforcement learning not being widely adopted as an AI tool for agents in games? Is it still too early to have this technology or are there technical limitations or practical challenges that make it difficult to implement? Or are other factors, such as ethical concerns or legal issues, hindering its adoption?

I would appreciate any insights or references to relevant literature or failed attempts to integrate reinforcement learning algorithms into games on this topic.

Answer 1

At a very high level: reinforcement learning solves a different problem than the one game AI programmers are typically trying to solve.

It's great if you want to make an opponent that's good at winning, efficiently in problem spaces that are incompletely known, possibly by exhibiting surprising strategies that a human wouldn't have intuited, thought to code into it, or know how to counter effectively.

In digital games, we often want our AI characters to play to lose, entertainingly, in problem spaces that are completely known (like a fixed map we can annotate with waypoints / behaviour hints in advance), and we want them to do it in ways that aren't overly surprising or counter-intuitive, else it looks like a bug or cheating.

This playing-for-fun-not-to-win is harder to quantify into a reward function, and it's difficult to anticipate and fence off every exploit or weird-looking emergent behaviour the RL process might wander into.

Sometimes we prefer AI who are downright predictable, because then our players have fun manipulating them, and our designers can more easily tune them to get the frequencies of different behaviours they want / that our playtests show tend to yield the most fun. (And as a bonus, our AI bugs are reproducible and easier to reason about and fix!)

Players who want to be challenged by an intelligent and adaptive opponent trying its best to beat them usually prefer to play against other human players. Letting them get their challenge fix that way builds social stickiness into the game by fostering play relationships with friends, guild mates, rivals, tournaments, streaming audiences... relationships players don't tend to form with AI opponents.

There could absolutely be benefit to having reinforcement learning opponents as a training mode to help someone get up to speed with competitive play, without the social shame of losing to fellow humans over and over. But the development cost of doing that well is probably quite high for many games, compared to using a ranking/matchmaking system to pair up beginner players with other beginners, as long as the game is able to maintain a critical mass of active players (and if it's not, the developers have bigger problems).

Answer 2

"Artificial intelligence" in games

In general, enemy agents in video games do not use machine learning. Instead they are expert systems, behavior tree, cellular automata, finite state machines, path-finding, and similar solutions. All of which were at some point considered artificial/computational intelligence.

These terms refer to always changing sets of technologies. For example, at some point path-finding was considered "Artificial intelligence". Currently when people say "Artificial intelligence" they are thinking of machine learning.

The rest of this answer is about machine learning and video games. Taking for granted that we can use machine learning models to assist in asset creation.

---

Making enemy agents with machine learning is not a great idea

A lot of the applications of machine learning to video games are because video games pose a complex controllable challenge for the artificial agents. So, it is the video game industry being useful for machine learning research, and not the other way around.

As DMGregory explains making opponent artificial agents fit into the desing of the game is a remarkable challenge, because we want them to "lose, entertainingly", and it would not be straight forward to train a model for that as opposed to training it to win. Also predictible enemies are often better as the players can steadily figure out how to defeat them.

Furthermore, machine learning solutions create friction to change (changes will often require to retrain the machine learning models) and are hard to debug (Explainable Machine Learning is an active area of research).

---

Examples of machine learning in video games

Again, taking for granted that we can use machine learning models to assist in asset creation.

A notable example of using machine learning in video games is augmented reality, which builds on top of computer vision. For example facial tracking and skeletal tracking implemented with pre-trained models.

We can find an examples of pre-trained models in more traditional games. For example, Supreme Commander 2 uses an artificial neural network pre-trained model to allow units to take decisions.

A more recent application is Deep Learning Super Sampling (DLSS). Yet, I'm betting that is not what you had in mind.

Furthermore, there are already experiments of using large language models and other generative models - running on a remote server - in games. But - at the time of writing - these are indie games that use it as gimmick.

---

Feasibility of training machine learning models during game play

So far we have talked about pre-trained models. That is because:

• Time: it is unlikely that your game session is not going to last long enough to train a model. Much less if you want to do learning at the character level, since your would be interacting with another character for just a small fraction of the game session (unless it is a character companion). And if it is an enemy character, chances are you kill it before you could tell if it was smart or not.
• Performance: You would be taking a fraction of the CPU time budget to train the AI. Remember that in video games the game already has a lot of things to do each frame to keep the game running smoothly.

Are there games that train a neural network during game play? Yes. In indie games where that is the point (games about creature evolution), or games created as part of research projects.

Addendum: There is a note worthy example: Black & White by Lionhead studios. It trained a neural network during game play, which was used as part of the pet creature behavior logic. The player could reward or punish the creature at any moment which would be applied to recent behaviors. Also, while leashed to the player the creature had a chance to mimic player actions. So positive training was the player would leash the creature, do the actions them wanted it to do, and then reward the creature for it. Negative was the player catching the creature doing something they didn't want to do and punishing it. I also want to point out that it was a very small model, which allowed for training it this way.

In fact, the situation might improve: The paper "The Forward-Forward Algorithm: Some Preliminary Investigations" released in December of last year introduces an alternative to back-propagation which is less computationally intensive and has faster training rounds (although it will require more training round to archive the same results) which might allow to train models during game play.

This does not change the fact that making opponent artificial agents work within the game design is a remarkable challenge. Yet, it makes other solutions more feasible, such as artificial agent companions that assist the player, and machine learning solutions for dynamic difficulty.