What algorithm can be used for audio segmentation? I want to divide song to segments and find the most energetic segments (begin and end in seconds). It is needed for the rhythm-game.


https://soundcloud.com/dieantwoord-yolandi/die-antwoord-cookie-thumper (most energetic segments: first 3:09-3:38; second 5:02-5:27)

https://www.youtube.com/watch?v=StZcUAPRRac (rammstein - sonne) (most energetic segments: first 0:55-1:20; second 1:52-2:44)

enter image description here


I think I have found an answer for this one.

You first load the audio data to an AudioClip of course. Then you sample the data from that clip into an array.

AudioSource aud = GetComponent<AudioSource>();
float[] samples = new float[aud.clip.samples * aud.clip.channels];
aud.clip.GetData(samples, 0);

Now, you have your data in hand. What you want to find out is the different segments of songs. So, I suggest (because this part can be done in numerous ways) going through the clip, X samples at a time, and getting the mean value of those samples.

Since samples vary between -1.0F and 1.0F, you do need to get rid of the negative values (or the positive whichever you prefer) for this method to be of any value.

int X = Y; //Y is the number of samples you want to consider, X is any name you want
int checkNum = samples.Length/X; //because we don't want this to be done a lot of times
float[] sampleMeans = new float[checkNum];
float currentMean = 0;
int j, i;
for (j = 0; j < checkNum; j++)
    currentMean = 0;
    for (i = 0; i < X; i++)
         if(samples[i+j*X]==null) //check whether the value is null 
             {break;} //and insure we don't go past the array
         if(samples[i+j*X]>=0) //make sure values are positive or taking a mean will not mean much (get it?)
             {currentMean = currentMean + samples[i+j*X];}
    sampleMeans[j] = currentMean / i;

After this is all done, you will have an array that represents the means of every X samples. You can check this array to see whether that part is more energetic.

For example if sampleMeans[2] is bigger than 0.5F, then you can say that the third X samples are energetic and start from samples[3*X] and iterate to samples[4*X-1] adding each element into a new array.

If the next part also suits your conditions, then add that part into the array you just made, if not, then make the last one into a new AudioClip by using SetData method. Then repeat the process with this new data set. (It's going to take some thinking, but it's all essentially gruntwork.)


You must log in to answer this question.

Not the answer you're looking for? Browse other questions tagged .