# How do audio based games such as Audiosurf and Beat Hazard work?

Note: I am not asking how to make a clone of one of these. I am asking about how they work.

I'm sure everyone's seen the games where you use your own music files (or provided ones) and the games produce levels based on them, such as Audiosurf and Beat Hazard.

Here is a video of Audiosurf in action, to show what I mean.

If you provide a heavy metal song, you would get a completely different set of obstacles, enemies, and game experience from something like Vivaldi.

What does interest me is how these games work. I do not know much about audio (well, data-side), but how do they process the song to understand when it is settling down or when it's speeding up? I guess they could just feed the pitch values (assuming those sorts of things exist in audio files) to form a level, but it wouldn't fully explain it.

I'm either looking for an explanation, some links to articles about this sort of thing (I'm sure there's a term or terms for it), or even an open-source implementation of this kind of thing ;-)

EDIT: After some searching and a little help, I found out about FFT (Fast Fourier Transform). This maybe a step in the right direction, but it is something that does not make any sense to me..or fits with my physics knowledge of waves.

-
I just edited to add some information on FFTs, hope that helps :) – Ray Dey Jul 10 '11 at 17:11

The term you're looking for is signal processing/analysis There are lots of techniques involved but the fundamental one that those games make use of is Beat Detection. This tries to calculate the tempo of the song and where the beats in a measure are and hence place the obstacles the appropriate distance apart to coincide with each beat.

The way that the games know when to "kick in" etc can range from being very simple and measuring the amplitude (volume) of the waveform or something more complex like isolating the volume of certain frequencies and measuring their volume.

If you're interested, look into Digital Signal Processing to see how you can analyse waveforms, which is essentially what these games are doing in their loading phase.

These links are good to get you started:

Hope that helps :)

-Ray

EDIT: I just saw your edit regarding Fourier transforms and thought I'll add some insight into it, although I'm by no means an expert on it!

FFT is a way of calculating the actual Fourier transform of a waveform. Basically, if you load up an audio file into Audacity, you'll see the wave form with the timeline along the top, this is known as the time domain. The FFT will convert a signal from the time domain into the frequency domain (basically all the frequencies that occur within the audio).

This conversion is useful for spectral analysis. In a game example, if you were to do a Fourier transform, you could easily calculate the amount of high frequency occurrences in the audio, and from that you could add twinkly visual effects, stars, or something associated with typically high frequency sounds. For the low frequencies you could have big, gluttonous monsters moving in time to the bass sounds, etc.

-
Great links! However, I clicked them and my eyes melted a little. :p – The Communist Duck Jul 9 '11 at 18:45
@The Communist Duck: That was the weak part. Now your eyes are stronger. – doppelgreener Jul 9 '11 at 23:29
@The Communist Duck You've just stepped further down the rabbit hole and it goes a looooooong way down ;) – Ray Dey Jul 9 '11 at 23:38
I'm kind of annoyed XNA doesn't really allow me to do any of these things with MP3 files (I could with .WAV I think, but those are huge) which is annoying. But hey, thank you very much for great resources and a headache :D – The Communist Duck Jul 11 '11 at 10:19
@Kay updated the link – Ray Dey Mar 4 '15 at 15:43

Here is a great seven part tutorial series on this topic by Badlogic Games. They cover everything from the basics to implementation.