How do I use music analysis in a game?

Question

I'm thinking of writing a game in the vein of BeatHazard (where the music you're listening to affects the game), and after researching the subject of music analysis in programming, I know enough to know that it's a very complex subject. So here are some questions:

1. What concepts do I as an engineer need to know about to do this? What concepts do I probably not need to know about?
2. What kinds of analysis are simple, robust, and efficient enough to be used in a game (and preferably in that order - I would greatly prefer simplicity over efficiency and can trade some correctness if it keeps things simple)?
3. What cross-platform open source libraries exist that are easy to build and include? I'm using MonoGame, but I'm willing to use p/invoke for a good library. I've read about aubio, clam, and marsyas, and I'm not quite sure which I'm looking for.

Answer 1

This is a very interesting question.

To analyze music, it is important to know exactly how one song differs from the next. Of course, music is a very abstract art field with a lot of incomplete and conflicting opinions and elements, but here is a short list of things that are generally agreed upon as making one song different from the next:

1. Tempo/Rhythm. Basically the speed of the song. How fast you are producing one sound after the other.
2. Pitch/Frequency. The (fundamental) frequency of one tone.
3. Timbre. The 'tone color'. Playing the same tone with the same pitch and length on a different instrument will create a different sound.
4. Amplitude. Basically how loud or soft a certain sound is played.
5. Harmony. When playing multiple sounds together or after oneother, the relation between their different pitches will change the feeling of the music.

The other thing you need to know about is how music is stored inside the computer. A music file consists of, for each sample in time (with many samples per second, 44.1 kilosamples or often higher) the amplitude of the speakers.

It might seem odd that only the amplitude over time is enough to store all four previously-mentioned elements, but this is natural: after all this is the same way sound is stored in the 'air' when it's traveling towards our ears.

How are these elements all related to amplitude?

1. Tempo/Rhythm: To put it very crudely the time between two tones is the time between two amplitude peaks. Rhythm is a sequence of these times repeating.

2. Pitch/Frequency, Harmony and Timbre: One amplitude changing over time can also be expressed as a multitude of Sine waves. By using a mathematic formula called the Fast Fourier Transform, computers do this (probably already behind your back) to give you visible harmonics and fun like http://en.wikipedia.org/wiki/Spectrograms. To put it simply:

   • Pitch is the fundamental frequency of one tone
   • Timbre are the 'overtones' of one tone
   • Harmony are the relations of different pitches occuring at the same time.

Obviously this is getting complex very quickly, and it is difficult to create robust systems that analyze the Pitch, Timbre and Harmony well(enough).

What kind of analysis are Simple, Robust and Fast - Amplitude/Harmony analysis. It is very straightforward to check how loud the song is(or how loud different frequencies of the song are) at a certain point in time. This is used all the time. Most audio visualisations are based around this approach.

• Timbre analysis. Check what kind of different overtones are loud right now. Depending on even/odd overtones(compare triangle with square and saw waves) you can say something about the general timbre of the sound (if you are hearing a 'lush' sound or a 'screechy' sound for instance).

About sound engines to use, I will get back to you. I will update and refine this answer shortly with that information. Sorry for the slightly long rant. I hope it might prove helpful.