4
\$\begingroup\$

Let us say I want to create a singing game similar to SingStar, and for this purpose I want to get the audio pitch the microphone is getting, in real time, or as close as possible.

I know I can get audio input in Godot. Even record the microphone to a file, there is an official demo project that does that (mic_record). I also know there are libraries that can analyze the audio.

How can I get the microphone pitch in real time?


Yes, I'm answering my own question. This question is inspired on a similar one on StackOverflow, which asked for a library to do this, which lead to the question getting closed, and the asker deleted it instead of editing the question… Before I noticed. As it turns out, you don't need libraries to do this. This is a means to answer the question I believe that person should have asked.

\$\endgroup\$
5
\$\begingroup\$

Getting Microphone Input

To have microphone audio input in Godot:

  • Go to Project -> Project Settings -> General -> Application -> Audio and make sure "Enable Audio Input" is On.
  • Go to the Audio panel (on the bottom of the Godot editor by default), and add a new Audio bus with a Record effect (AudioEffectRecord), enabled.
  • Add an AudioStreamPlayer to the scene tree. Set the Stream attribute to a New AudioStreamMicrophone, and the Bus attribute to the Audio bus you created. We also want it to be playing, so put set the Autoplay attribute On.

If everything is working, the game should be playing whatever is listening from microphone. Once you have made sure it is working, mute the Audio bus.

The official demo project mic_record shows how to record and play the audio, and how to save it to a file. We don't need any of that.

No code required.

By the way, Godot includes other useful effects including equalizers and a pitch shifter.


See also:


What is Pitch?

What we refer as pitch is the audio frequency. However, is there a particular pitch for an audio signal? We need to consider that real audio is a mix of multiple frequencies.

Even an instrument playing a single node will have a fundamental frequency, and overtones which are weaker than the fundamental. (See Harmonic). Thus, we would want the stronger frequency.


Well, turns out we cannot get a specific frequency anyway. The best we can do is get frequency range. So pick reasonable ranges to query. Perhaps based on musical notes (you would want the nominal frequency of the note at the center of the range).

The problem is that the more certain we are of the instant in time we want to measure, the less certain we are about the frequency of the audio. Yes, that is the uncertainty principle. It emerges from the Fourier transform we need to do to get the frequencies.

And I'm not going to explain that, instead I'll recommend the video But what is the Fourier Transform? A visual introduction. by 3Blue1Brown, and its follow-up video he more general uncertainty principle, beyond quantum. Also remember we have audio samples to work with, not a function.


Spectral Analysis

As it turns out, we don't need any library to do a Fourier Transform in Godot, because it already has the means to do an spectral analysis.

To do that add a SpectrumAnalyzer effect (AudioEffectSpectrumAnalyzer) to the Audio bus you want (the same with the Record effect, in this case). It will do a Fast Fourier transform for us.

You can retrieve an AudioEffectSpectrumAnalyzerInstance from code using AudioServer.get_bus_effect_instance, and call get_magnitude_for_frequency_range on it.

The method takes a frequency range and returns a Vector2 (as far as I can tell - this is not documented at the time of writing - these are the left and right stereo components※, for a fair combination use the length of the vector) in linear energy scale (you can use linear2db to get decibels).

To get the stronger frequency range, we call the method on a loop changing the frequency ranges, and pick the maximum.

The method is also useful for audio visualization.


See also:


※: Yes, a Fourier Transform has a real an an imaginary part. However, after having a look at the source code, I don't think that is what is going on. The source for method (here) uses a fft_history array. Which we see (here) converted to audio frames. Which have l and r (as defined here). Unless that means "limaginary" and "real", I think they are "left" and "right".

\$\endgroup\$
3
  • \$\begingroup\$ great answer! I may have to mess around with this in godot and see what cool stuff I can come up with. It's nice how much work godot does for us. \$\endgroup\$
    – Millard
    Apr 29 at 19:55
  • \$\begingroup\$ This is awesome! Could you please elaborate on how to get the strongest frequency, though? Also, is there a way to handle multiple microphones? \$\endgroup\$
    – EzPizza
    May 31 at 16:29
  • 1
    \$\begingroup\$ @EzPizza It is possible to change the audio capture device, but not to have multiple (which is fine for most platforms and most games, but sadly not for couch co-op karaoke). It is probably worth a proposal. About the frequencies, you query a range (get_magnitude_for_frequency_range has from_hz and to_hz parameters). The official demo makes an audio visualization by querying multiple ranges. It is not hard to imagine picking the higher one. You can do a bit better by testing subranges (e.g. find the highest octave, then find the highest note on that octave). \$\endgroup\$
    – Theraot
    May 31 at 16:44

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.