Neural Net Controlled Car

Question

I'm trying to make an AI car controlled by a neural net.

I saw this two videos: Neural Network Demo and Q Learning and neural network in 2D car driving and I want to replicate that.

I already have the neural net code made, with a back propagation algorithm.

The thing is, I don't know how to reinforce the learning of the net. What kind of value should I use to calculate the error?

My car currently have 5 inputs (similar to the first video), and outputs 2 numbers, 1 which is plugged in the rotation torque of the car, if it's positive it will rotate clock-wise, if it's negative counter-clock-wise, and how much it will turn is based on it's magnitude, it ranges from -1~1, and I mapped it to a desired min-max rotation. The second output number is acceleration, it's from 0~1, but mapped to -10~100 (so it can reverse).

Answer 1

In general you want to give the neural net a reward function that it is trying to maximize, or, alternatively, a cost function it is trying to minimize. In the car example, "distance along the track" is a good indicator of progress. You can simply reward the neural net for making progress along the track.

In the simple case, imagine a neural net that tries to drive a car along a perfectly circular track with radius R, centered at [0, 0]. It has only one output to the car: steering angle (a), and only two inputs: the angle of the car around the track (t), and the radius from the center (r). Assume the car has a constant forward velocity of 1. Assume the car moves for a fixed number of timesteps N.

Therefore, given an action a_i, we can produce a state [t_(i + 1), r_(i + 1)] as:

x_i = [cos(a_i), sin(a_i)] + [r_i * cos(t_i), r_i * sin(t_i)];
f(a_i, t_i, r_i) = [atan2(x), norm(x)];

We can create a simple reward function as telling it to advance along the circle and maintain a constant radius:

R(t, r) = t + (R - r)^2;

The car tries to maximize the sum of this reward over its entire trajectory with its policy:

Sum(i = 1 to N of R(t_i, r_i))

So, the neural network generates a policy based on the state of the car. This in turn produces a trajectory, which is scored based on the reward function. The neural net attempts to learn a policy that maximizes the reward function in expectation over time. That is, it is trying to find:

policy*(t_i, r_i) = 
  arg_max Sum(i = 1 to N of R(f(policy(t_(i - 1), r_(i - 1)), t_(i - 1), r_(i - 1)))