2
\$\begingroup\$

I'm trying to make an AI car controlled by a neural net.

I saw this two videos: Neural Network Demo and Q Learning and neural network in 2D car driving and I want to replicate that.

I already have the neural net code made, with a back propagation algorithm.

The thing is, I don't know how to reinforce the learning of the net. What kind of value should I use to calculate the error?

My car currently have 5 inputs (similar to the first video), and outputs 2 numbers, 1 which is plugged in the rotation torque of the car, if it's positive it will rotate clock-wise, if it's negative counter-clock-wise, and how much it will turn is based on it's magnitude, it ranges from -1~1, and I mapped it to a desired min-max rotation. The second output number is acceleration, it's from 0~1, but mapped to -10~100 (so it can reverse).

\$\endgroup\$
  • \$\begingroup\$ I've seen good results using genetic algorithms to find good neural networks for car driving. An alternate method to backpopogation and other learning algorithms. \$\endgroup\$ – Alan Wolfe May 13 '15 at 21:55
  • \$\begingroup\$ A buddy of mine did his MSc thesis on Q-learning for a racecar. Maybe this will help you. \$\endgroup\$ – Junuxx Nov 10 '15 at 21:19
2
\$\begingroup\$

In general you want to give the neural net a reward function that it is trying to maximize, or, alternatively, a cost function it is trying to minimize. In the car example, "distance along the track" is a good indicator of progress. You can simply reward the neural net for making progress along the track.

In the simple case, imagine a neural net that tries to drive a car along a perfectly circular track with radius R, centered at [0, 0]. It has only one output to the car: steering angle (a), and only two inputs: the angle of the car around the track (t), and the radius from the center (r). Assume the car has a constant forward velocity of 1. Assume the car moves for a fixed number of timesteps N.

Therefore, given an action a_i, we can produce a state [t_(i + 1), r_(i + 1)] as:

x_i = [cos(a_i), sin(a_i)] + [r_i * cos(t_i), r_i * sin(t_i)];
f(a_i, t_i, r_i) = [atan2(x), norm(x)];

We can create a simple reward function as telling it to advance along the circle and maintain a constant radius:

R(t, r) = t + (R - r)^2;

The car tries to maximize the sum of this reward over its entire trajectory with its policy:

Sum(i = 1 to N of R(t_i, r_i))

So, the neural network generates a policy based on the state of the car. This in turn produces a trajectory, which is scored based on the reward function. The neural net attempts to learn a policy that maximizes the reward function in expectation over time. That is, it is trying to find:

policy*(t_i, r_i) = 
  arg_max Sum(i = 1 to N of R(f(policy(t_(i - 1), r_(i - 1)), t_(i - 1), r_(i - 1)))
\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.