Is there a PBR-like material system for audio?

Question

With visual/rendering materials, a very popular system for this is PBR. Materials made using PBR may define properties like albedo color, normal maps, emission, roughness, metallic, specular, etc. This allows creating objects that interact with light in a realistic way, using general-purpose parameters.

With physics materials, it's common for them to define friction, restitution/bounciness, absorbency, etc. This allows defining how objects react to each other when sliding against each other or impacting each other in the physics engine, using general-purpose parameters.

With audio materials, such as those used for calculating footstep sounds, impact sounds, bullet hole sounds, etc, as far as I know most games just use a hard-coded list of sounds. For example a game may define an object as sounding like "wood", so you walk on that object and it plays the wood footstep sound.

However, I am interested in building an engine-agnostic representation of audio materials, so a hard-coded list of sounds is not ideal. I would prefer to have a series of parameters like PBR provides for visual materials. For example if game engine A has "wood" and "birch wood", and game engine B has only "wood", I would like to be able to convert "birch wood" into parameters, then game engine B can play the closest equivalent sound to those parameters, or possibly interpolate between sounds, or procedurally generate sounds.

Is there a PBR-like material system for audio? Is there a way to define how objects sound using a series of parameters instead of a hard-coded list of strings? Some set of parameters to define audio resonance, brittleness, impact response, thermal response, moisture response, hollowness, reverberation, etc? Do any existing game engines have this? What about other software like those used to make movies?

Answer 1

There is something analogous to physically based rendering for audio: physical modelling synthesis.

However, while you will find models that contain more or less simple parameters, you will not find one model that works — you will not find a direct equivalent of the visual “PBR material” concept. This is because the problem of producing a complete realistic sound is analogous to the problem of producing a complete realistic image: you don't just need one material, but a “scene” in which sound waves propagate and are changed.

To produce an image, we use a rendering algorithm and a prepared data set which encodes decisions about what light rays and what surface details are worth calculating. To produce a sound, we need to decide what signal-processing elements are worth calculating, and what the graph of interactions between them is.

We can choose to produce an image by brute force using a path tracing renderer without the approximations that are used for real-time graphics, and trade off efficiency for generality and realism. Similarly, we can produce a sound by creating a model of the physical structure of the vibrating elements, and running a finite element method calculation that simulates how the vibrations propagate through the object(s) and out through the air. But this is not cheap, especially because your simulation step rate cannot simply be the 30-120 Hz that satisfies human vision, but rather 40+ kHz. (This has been done experimentally in real time, using input from a physics engine to excite the finite element mesh and thus producing physically-based collision sounds; unfortunately I don't have a link to the paper as I happened across it many years ago.)

It is possible to write down reusable descriptions of much simpler synthesis systems — the graph of signal processing elements I mentioned previously. This is the closest analogue of a “PBR material” in the audio synthesis world. However, I am not aware of very much standardization in this area; the possible nodes and parameters are usually unique to a specific tool. One exception I'm aware of is the Web Audio API being a specification rather than a program; however, it doesn't come with a way to define its graphs as data rather than programs.

Another factor is that the things you can readily reproduce using simple synthesis elements are generally not similar to complex everyday sounds — rather, they tend to be excessively musical. This is analogous to how a visual material will involve textures, in order not to look like a too-perfect plastic or metal shape.

Similarly, you can start with one or more sound samples and manipulate them in physically based ways. The most common form of this is spatial audio: besides just volume and panning, you can have:

• frequency-dependent attenuation, dependent on the frequencies the materials in the environment absorbs preferentially
• Doppler effect simulation for moving objects
• reverberation for the enclosing room

(Note that these are all approximations of specific elements the true propagation of sound, just like much of real-time graphics is approximation of the propagation of photons, and just like graphics, you can use a ray-tracing approach to better model propagation, but it requires more computation.)

Beyond those, there are also ways to parameterize processing of a sound sample to make it useful for more purposes, such as objects of different sizes or related materials:

• Pitch shifting for objects of different sizes
• Apply a filter (low-pass, high-pass, equalization…)
• Granular synthesis: play many overlapping short sounds, and adjust the timing, pitch (or rather, playback rate), density, and mix of multiple samples to produce a continuously variable, complex output.

However, all of this will still usually be done starting with specific sound samples for the application. You could hope to find matching samples that suit your needs, and shape them using filters, but I think in practice you should expect to bring your own sounds just like you bring your own textures. You might pick them out of an asset library, but you'll still be picking them, not just getting an automatic match from a physical description. Unfortunately.

Your idea of going from “wood” to “birch wood” automatically would be difficult for two reasons: first, there is not any standard for what the “generic wood impact sound” would be like (in frequency spectrum before any added filters), and second, the sound will be very different depending on what the actual wood structure involved is (a tree? A deck? A wooden box?) because that affects the resonances. You can use parameters to stretch a few sounds to cover many more cases — it's just that there is no one unified “PBR sound material”, but a bunch of arbitrary choices.

In summary: Expect to treat sound as an art creation problem, not something the engine can do for you.