Using spherical harmonics to build complex sound scapes

Question

I was having a discussion at work today on how to improve room acoustic modelling. It was suggested that very little work appears to have been done on the subject yet tonnes has been done in graphics. It was further suggested that as sound and light are just wave forms it might be possible to use things like ray tracing and radios to get a good approximation.

At this point I began thinking about spherical harmonics to simulate this behaviour. You could simultaneously do specular-esque reflections as well as audio scattering modelled as diffuse reflections. As a bonus you'd also get obstructions blocking sound transfer. The only problem was how to handle the attenuation of multiple frequencies. Surely though this could be modelled by applying some form of transform to the audio spectrum (FFT).

Anyway does anybody know of any papers on this subject, especially on how it would be performed in real-time. Failing that anybody got any advice or useful general information?

Answer 1

Yeah, it's a good idea. Malham already wrote a paper on this, which was based on some comments by Menzies (back in 1999!).

Also note Nosal's MASc thesis discussing using radiosity for acoustics.

As for lighting, (which does 3 sets of functions, one for R, one for G, one for B) you would need to do a different "set" of SH functions for each rough frequency band you want to represent (say one for low frequency, 60Hz-1kHz, one for mid 1kHz-2kHz, one for high 10kHz-20kHz, etc). Let everything below 60Hz pass through everything (which is pretty much what sound does in real life)

You'd need to model the acoustic reflection capabilities of each material, however, just as lighted materials each respond to RGB differently.

Answer 2

It sounds like an interesting problem, though I am wondering how much accuracy you will need before people just can't tell. Anyway, this answer focuses on the "real time" part, but I don't know of any papers as it isn't something I've investigated.

Obviously if you want to calculate this accurately in real time, with mobile sound sources, calculating reflection would be fastest using the graphics card. For example if you have a simplified version of the world running simultaneously, you could use it to render "reflection patterns" to a texture or cubemap and infer from that texture how you should output the sound. Sounds (or separate frequency bands of sounds) in that model would be point light sources. With only basic (1 bounce) reflections you might find that you don't need any more accuracy anyway, while this should be very quick especially with simplified geometry and reduced resolution. I'm not entirely sure if there are performance problems with multiple scenes on one graphics card, though.

Getting further into areas I know little about, a BSP tree seems like it could be useful for waves that bend around corners as (I think) it defines volumes and their connections to other volumes.

Depending on the situation, a further optimisation would be to cache results from the above tests. For example, storing a cubemap of sound that could be rotated based on the players orientation but not recalculated entirely, or a couple of cubemaps that you can interpolate between based on the player's position.

Answer 3

I saw a talk at GDC on this 6 or 7 years back. Possibly this one: Geometric Modeling of Sound Propagation in 3D Games.

Others by the same author:

• Priority-Driven Acoustic Modeling for Virtual Environments
• Interactive Acoustic Modeling for Virtual Environments

Answer 4

I haven't actually tried this, but I've always wondered if an approach such as light propagation volumes could be used for audio. In light propagation volumes a small 3d texture is used (32x32x32 I believe) in which light bounces and occlusions are simulated by flood filling the 3d texture. Since it uses spherical harmonics, it might be able to do this with audio as well. I'm not an audio expert however

Answer 5

Doug James of Cornell's computer graphics program has done a lot of work in accurate modeling of sounds in an environment. However, most of his papers handle specific cases of sound generators (thin shell objects, flames, etc.) They also are probably not efficient enough to do in real-time along with the other tasks your game has to do.

However, it may be of use for you to read through some of them. It may give you ideas on how to proceed and/or modify his approaches to be cruder but efficient enough for real-time performance.

His site is here:

http://www.cs.cornell.edu/~djames/

Of particular interest might be his "Harmonic Fluids" and "Harmonic Shells" papers.

Answer 6

I have also given this some thought. I felt the primary concern (in terms of realism/performance trade-off) was that spatially your ears are inferior to your eyes - and quite easily accept something that might not be as realistic as your eyes would need. There is a very good chance that trying to perfectly model sound in the local environment is overkill - EAX is probably 'good enough'.

In a closed environment (e.g. Quake) I would firstly calculate two properties about each room: 'transferrance' and immersion:

Transferrance would indicate how the sound would be affected by travelling through this room and would most likely count toward a parametric EQ (ideally you would add echo/reverb from each room, but your EAX chip might not have this much bandwidth). The parametric EQ would also ultimately simulate the sound attenuation.

Immersion would be calculated by splitting the room into nine cubes (possibly, even just one might be good enough) and calculating the local sound properties from that perspective. These parameters would be used in the EAX environment.

Finally each of your rooms would be connected by a graph, where each point in the graph are the portals connecting each room.

When the sound triggers you would do a flood fill (no an A* search) and keep track of the the transferrance and distance travelled. When the sound reaches the player you would queue it up to play at some point in the future; based on the distance travelled. You might keep track of the number of graph points passed and eventually 'cull' the sound (in other words, a continuous flood fill). You might have to use CUDA to do this as it could get CPU bound.

When a sound plays you would use a 3D sound API (OpenAL) to and place it at the portal at which it entered through, you would then find out which of the nine cubes the player is currently in and apply that EAX environment.

The neat thing here is that if your environment is sufficiently complex you would get free 'global' echoes and players would perceive sound coming from the right direction; and if you get the EAX environment correct hopefully the effect would be convincing enough that the brain would accept it.