Take the 2-minute tour ×
Game Development Stack Exchange is a question and answer site for professional and independent game developers. It's 100% free, no registration required.

I am looking to use reinforcement learning to adaptively modify the weights involved in a flocking algorithm (i.e. 'boids').

Searching google revealed several libraries, but I don't know anything about any of them.

I am far from being an expert in reinforcement learning, though I have perused Sutton and Barto's classic book on the subject, so I understand some of the concepts.

A little background, in case it helps. I have a few steering behaviours (seperation, alignment, cohesion, forward, and wall avoidance), which each generate a 'force' vector and compete to tell the agent(s) where to move. Each force vector is combined in, basically, a weighted sum. I would like to 'learn' these weights adaptively so as to cause e.g. more/less clustering, reduce/increase the flock's speed, etc.

Can anyone recommend a good library to use?

I would also need some kind of linear classifier, or something, to interface the learning with the (float valued) weights for each of the flocking steering behaviours. So a library that provides a one-stop solution for this problem would be good?

share|improve this question
1  
How do you intend on training the birds? That is probably the biggest point, there are plenty of intelligent systems you can input the data into; but the one that gives you what you want with the least trouble is what you want. So, how do you intend on training them? How are separating, the good from the bad. How you answer this question will determine what library you'll probably end up using. –  Daniel Apr 14 '11 at 13:49
    
For example, I could use a reward function based on speed and average distance to cause the flock to cluster together and stop moving (or move very slowly). Then, alter the reward function again to make the cluster separate and begin to flock forward in formation. –  Dave Apr 14 '11 at 14:30
add comment

2 Answers

As Daniel pointed out, the difficult part is the reward function. If you got that figured out what it comes to to is this:

  1. You create a flocking simulation taking N parameters (where N is the number of coefficients to tune) returning a single value (the "fitness/error/reward").
  2. Afterwards you plug that function into a reinforcement-learning algorithm as a black-box error-function.
  3. The algorithm will then try more or less random values for the N parameters, evaluating their error (by launching your simulation), and stop when good values are found.

A very fast and stable reinforment learning algorithm is the CMA-ES. Papers and source code are available at http://www.lri.fr/~hansen/cmaesintro.html

Be warned though:

  • Building a reward function that actually does what it should is often more difficult than solving the initial problem manually.
  • The reward function should return the same output value for same input values, therefore "then alter the reward function" is probably a bad idea.
share|improve this answer
add comment

During my software agents module, I searched high and low for a decent RL learning. Unfortunately, I didn't find anything that was a) useful for my uses and b) well documented.

However, something like Q-Learning isn't too difficult to implement. Artificial Intelligence for Games has a simple implementation of Q-Learning that would probably be of use to you. What's more is the source is packaged on the CD. I would have suggested the website for the book to get the source code but it seems to be down or just gone.

Also, here is a reinforcement learning simulator (open source and written in Java) which is fairly straightforward to understand the underlying concepts. http://www.cs.cmu.edu/~awm/rlsim/

But as said above, the reward function you use is key to getting a decent result.

Personally, when it comes to tweaking weights, I'd probably use a genetic algorithm, but similarly you'll need the appropriate fitness function for what you're trying to create.

share|improve this answer
add comment

Your Answer

 
discard

By posting your answer, you agree to the privacy policy and terms of service.

Not the answer you're looking for? Browse other questions tagged or ask your own question.