How to train a reinforcement learning algorithm in Unreal?

Question

I know I can use ONNX plugins to use pretrained Deep Learning models in Unreal Engine 5, like style transfer, object detection, etc.

I want to build a basic reinforcement learning project and would be happy to code it from scratch like a q-learning algorithm, but I am finding it is quit difficult to do that. I saw that we can convert Python PyTorch scripts into .onnx format to use it in Unreal Engine (C++), but there is no documentation I found on how to use .onnx files in UE5 or 4.

I am trying to build a model for a basic explore and exploitation task. A character will be there on the floor and an object will be there. The character has to find a path on the floor to reach the object (terminal state).

Answer 1

Starting in UE 5.3 they added Learning Agents, a plugin that allows you to train agents via reinforcement and imitation learning. See here for more.

Side note: using reinforcement learning for path finding is usually not recommended. Read this Stack Exchange question & answers for more.

Answer 2

The short answer is not directly.

Unreal is not designed for ML training tasks.

You can collect a dataset by logging an actor's movements and the notable collisions for training, and train on that data externally.

The A* algorithm and the built-in Navigation system make reinforcement AI learning for navigation unneeded. Given the bias of the limited sample size and lack of diversity, the generic ML application of the data collected would be mostly useless.

Answer 3

There is a plugin called MindMaker which we can use to build reinforcement learning models in Unreal Engine itself.