18
\$\begingroup\$

Reinforcement learning has shown remarkable success in game-playing agents, as seen in the achievements of AlphaGo and OpenAI's Dota 2 AI. There are a few tools, such as Unity ML Agents, to integrate state-of-the-art algorithms into game engines, and we have recently seen that Grand Turismo 7 added an AI racer trained with reinforcement learning into their game. However, this approach to creating intelligent agents seems not widely adopted in many other complex games where AI agents such as NPCs and other opponents could benefit from its capabilities. Examples of successful integrations are still minimal.

I'm curious to know the reasons behind this trend. Why is reinforcement learning not being widely adopted as an AI tool for agents in games? Is it still too early to have this technology or are there technical limitations or practical challenges that make it difficult to implement? Or are other factors, such as ethical concerns or legal issues, hindering its adoption?

I would appreciate any insights or references to relevant literature or failed attempts to integrate reinforcement learning algorithms into games on this topic.

\$\endgroup\$
9
  • 2
    \$\begingroup\$ You may be interested in past Q&A about this, including Is reinforcement learning overkill for a vehicle steering bot? Why is reinforcement learning so rarely used in pathfinding? and Choosing AI strategy. \$\endgroup\$
    – DMGregory
    Commented Mar 8, 2023 at 20:09
  • 2
    \$\begingroup\$ Imagine an MMORPG where a raid boss would learn over the weeks. Waiting for the pull? Alphastrike for the win. You have tanks in the front? Why would I go for those, that healer at the back three weeks ago was dancing on my corpse. Why should I only use my weapon when I'm a 50m tall statue and can just stomp one you? \$\endgroup\$
    – Zibelas
    Commented Mar 9, 2023 at 15:40
  • 3
    \$\begingroup\$ @Zibelas That would be awesome. Can you imagine the drama created by a boss that doesn't get numerically stronger, just thinks better? If its loot increased based on the time since it was last beaten, that could be a huge draw for a MMORPG all on its own. \$\endgroup\$ Commented Mar 9, 2023 at 21:00
  • 1
    \$\begingroup\$ @silvado That's very simple for sure, but that's not how you get a slightly dumber AI - you get a good AI most of the time and an idiot the rest of the time. It doesn't really decrease the difficulty unless you're already right at the margin (except for trivial games like Diablo). It also feels really stupid. Getting fun out it is quite hard. \$\endgroup\$
    – Luaan
    Commented Mar 10, 2023 at 10:21
  • 1
    \$\begingroup\$ @silvado For chess and similar games, a decent human player doesn’t generally make perfect moves some percentage of the time and then mediocre moves the rest of the time, they make reasonably good moves most of the time, with a few really good moves, and a few blunders mixed in. That ‘most of the time’ is what actually dictates how difficult a given opponent is, but it’s also extremely challenging to balance for things that cannot be reduced to raw numbers like chess (or xiangqi, or go, or shogi) can be because it’s not possible to make an absolute measure of how good a given choice is. \$\endgroup\$ Commented Mar 10, 2023 at 17:23

2 Answers 2

54
\$\begingroup\$

At a very high level: reinforcement learning solves a different problem than the one game AI programmers are typically trying to solve.

It's great if you want to make an opponent that's good at winning, efficiently in problem spaces that are incompletely known, possibly by exhibiting surprising strategies that a human wouldn't have intuited, thought to code into it, or know how to counter effectively.

In digital games, we often want our AI characters to play to lose, entertainingly, in problem spaces that are completely known (like a fixed map we can annotate with waypoints / behaviour hints in advance), and we want them to do it in ways that aren't overly surprising or counter-intuitive, else it looks like a bug or cheating.

This playing-for-fun-not-to-win is harder to quantify into a reward function, and it's difficult to anticipate and fence off every exploit or weird-looking emergent behaviour the RL process might wander into.

Sometimes we prefer AI who are downright predictable, because then our players have fun manipulating them, and our designers can more easily tune them to get the frequencies of different behaviours they want / that our playtests show tend to yield the most fun. (And as a bonus, our AI bugs are reproducible and easier to reason about and fix!)

Players who want to be challenged by an intelligent and adaptive opponent trying its best to beat them usually prefer to play against other human players. Letting them get their challenge fix that way builds social stickiness into the game by fostering play relationships with friends, guild mates, rivals, tournaments, streaming audiences... relationships players don't tend to form with AI opponents.

There could absolutely be benefit to having reinforcement learning opponents as a training mode to help someone get up to speed with competitive play, without the social shame of losing to fellow humans over and over. But the development cost of doing that well is probably quite high for many games, compared to using a ranking/matchmaking system to pair up beginner players with other beginners, as long as the game is able to maintain a critical mass of active players (and if it's not, the developers have bigger problems).

\$\endgroup\$
9
  • 16
    \$\begingroup\$ Hyperparameters make sense for reinforcement learning, but as I described, we're usually not using reinforcement learning, so they don't apply here. Designers will often have more direct control to say "this NPC should patrol this exact path, pausing every 15m to do a flashlight sweep. If they see something they have an x% chance to flinch and a y% chance to go straight to combat state..." and implement a state machine that does exactly those steps. Then they can tune the non-hyper-parameters of durations, distances, speeds, transition probabilities, waypoint positions, etc. \$\endgroup\$
    – DMGregory
    Commented Mar 9, 2023 at 16:02
  • 2
    \$\begingroup\$ Another advantage of having NPCs behave predictably is that it means that even if real-time gameplay involves one player against the machine, consistent behavior makes it possible to have people compare their performance against other players, making it possible to compete against a much wider field than even the largest multi-player real-time games. \$\endgroup\$
    – supercat
    Commented Mar 9, 2023 at 21:04
  • 13
    \$\begingroup\$ "My AI started exploiting physics glitches to kill the player!" \$\endgroup\$
    – NotThatGuy
    Commented Mar 10, 2023 at 9:37
  • 3
    \$\begingroup\$ As a side note, I forgot what game it was exactly, but during the testing stages they noticed they programmed the AI way too good, i.e. they would silently surround you and you stood no chance in the ensuing firefight, causing the developers to dumb the AI down. Alongside games, movies also share the trend that there's a hero who fights against nearly impossible odds. This basically demands imperfection if not downright stupidity, sometimes beyond suspension of disbelief, of the opposing force. BondVillainStupidity \$\endgroup\$ Commented Mar 10, 2023 at 16:30
  • 2
    \$\begingroup\$ @infinitezero: I remember reading something about that regarding the F.E.A.R. games, if I remember correctly; one of the problems they had was not just having their A.I. decisions making good decisions, but having the squad of enemy soldiers "Talk" about their decisions, in a way that was clear, audible, and importantly....after the A.I. made the decision to take a flanking position, but before the A.I. actually gets around to flanking the player. Which means your A.I. has to know what the maneuver they're about to performed would be called in a way that's recognized by a human opponent. \$\endgroup\$ Commented Mar 10, 2023 at 23:24
12
\$\begingroup\$

"Artificial intelligence" in games

In general, enemy agents in video games do not use machine learning. Instead they are expert systems, behavior tree, cellular automata, finite state machines, path-finding, and similar solutions. All of which were at some point considered artificial/computational intelligence.

These terms refer to always changing sets of technologies. For example, at some point path-finding was considered "Artificial intelligence". Currently when people say "Artificial intelligence" they are thinking of machine learning.

The rest of this answer is about machine learning and video games. Taking for granted that we can use machine learning models to assist in asset creation.


Making enemy agents with machine learning is not a great idea

A lot of the applications of machine learning to video games are because video games pose a complex controllable challenge for the artificial agents. So, it is the video game industry being useful for machine learning research, and not the other way around.

As DMGregory explains making opponent artificial agents fit into the desing of the game is a remarkable challenge, because we want them to "lose, entertainingly", and it would not be straight forward to train a model for that as opposed to training it to win. Also predictible enemies are often better as the players can steadily figure out how to defeat them.

Furthermore, machine learning solutions create friction to change (changes will often require to retrain the machine learning models) and are hard to debug (Explainable Machine Learning is an active area of research).


Examples of machine learning in video games

Again, taking for granted that we can use machine learning models to assist in asset creation.

A notable example of using machine learning in video games is augmented reality, which builds on top of computer vision. For example facial tracking and skeletal tracking implemented with pre-trained models.

We can find an examples of pre-trained models in more traditional games. For example, Supreme Commander 2 uses an artificial neural network pre-trained model to allow units to take decisions.

A more recent application is Deep Learning Super Sampling (DLSS). Yet, I'm betting that is not what you had in mind.

Furthermore, there are already experiments of using large language models and other generative models - running on a remote server - in games. But - at the time of writing - these are indie games that use it as gimmick.


Feasibility of training machine learning models during game play

So far we have talked about pre-trained models. That is because:

  • Time: it is unlikely that your game session is not going to last long enough to train a model. Much less if you want to do learning at the character level, since your would be interacting with another character for just a small fraction of the game session (unless it is a character companion). And if it is an enemy character, chances are you kill it before you could tell if it was smart or not.
  • Performance: You would be taking a fraction of the CPU time budget to train the AI. Remember that in video games the game already has a lot of things to do each frame to keep the game running smoothly.

Are there games that train a neural network during game play? Yes. In indie games where that is the point (games about creature evolution), or games created as part of research projects.

Addendum: There is a note worthy example: Black & White by Lionhead studios. It trained a neural network during game play, which was used as part of the pet creature behavior logic. The player could reward or punish the creature at any moment which would be applied to recent behaviors. Also, while leashed to the player the creature had a chance to mimic player actions. So positive training was the player would leash the creature, do the actions them wanted it to do, and then reward the creature for it. Negative was the player catching the creature doing something they didn't want to do and punishing it. I also want to point out that it was a very small model, which allowed for training it this way.

In fact, the situation might improve: The paper "The Forward-Forward Algorithm: Some Preliminary Investigations" released in December of last year introduces an alternative to back-propagation which is less computationally intensive and has faster training rounds (although it will require more training round to archive the same results) which might allow to train models during game play.

This does not change the fact that making opponent artificial agents work within the game design is a remarkable challenge. Yet, it makes other solutions more feasible, such as artificial agent companions that assist the player, and machine learning solutions for dynamic difficulty.

\$\endgroup\$
5
  • \$\begingroup\$ Thank you for the explanation about the AI part of game development. I wanted to narrow the question to reinforcement learning algorithms (policy based or similar). Why would it be challenging to design an agent for losing against them while still being entertained? "Are there games that train a neural network during game play? Yes. In indie games where that is the point (games about creature evolution), or games created as part of research projects." Could you share an example? \$\endgroup\$
    – rcmalli
    Commented Mar 9, 2023 at 14:01
  • \$\begingroup\$ @rcmalli There are applications that train agents on a virtual environment. You may find indie example with a search. A somewhat common game design that does not have the player idly watching is to have the player design the creatures. And you know what, now that you ask for example, there is a game from a well known studio that slipped my mind while redacting the answer: Black & White from Lionhead Studios, it trained a neural network during game play which was used as part of the process to pick the creature behavior (which the player can reward or punish). \$\endgroup\$
    – Theraot
    Commented Mar 9, 2023 at 14:22
  • 1
    \$\begingroup\$ @rcmalli Anyway, about training an opponent agent: Winning is an easy to check, by understanding the problem space we might be able to make a metric for how close is an agent to winning. Yet, for the opponent the goal is not for it to win, but to provide an entertaining experience to the player, which is hard to check, much harder to measure, and you want to train the agent on that. I cannot argue that to be impossible. I remember something about using facial emotion detection during play testing, but I can't find the article. So, perhaps it is doable, but it is a remarkable challenge. \$\endgroup\$
    – Theraot
    Commented Mar 9, 2023 at 14:24
  • \$\begingroup\$ SC2 used a pretrained moel for their AI? And that game is ancient lol \$\endgroup\$
    – Hobbamok
    Commented Mar 9, 2023 at 23:06
  • 1
    \$\begingroup\$ One notable exception would be games like Chess or even Arcomage, where the game isn't pre-scripted but really about beating a tough opponent. We've actually used reinforcement learning in a game like that sometime around 2003 I think? It's not like it's a new technique, it's just that there's quite a few relatively high-profile recent examples of success. And even then, while we did the training with real players, the final game had it disabled and just used the pre-trained models (with different difficulties). \$\endgroup\$
    – Luaan
    Commented Mar 10, 2023 at 10:26

You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .