4
\$\begingroup\$

I have a project whose subject is "Sound Synthesis from object collision in Virtual Reality Environment" First of all, I have to define some objects with certain attributes (like shape, mass, material) and then I have to create sound from the collision of these objects in C++ according to their attributes, area of collision etc. I think I have to use some sort of Physics Engine like Bullet or ODE but I'm not sure how to start this project. Is there any Physics Engine specialized in Sound Synthesis?

I've come up with PhyA but I don't know how to download it and implement it. http://www.tech.dmu.ac.uk/~dylan/z/phya/

P.S. I intend to use CHAI3d or H3dApi because I have to use the Falcon device.

\$\endgroup\$
  • \$\begingroup\$ I'm sorry, but unfortunately we don't allow "which technology to use" questions because they end up in opinionated debates, not in indisputably correct answers. Also, I don't think that this subject is really on-topic here, because I don't think I have ever seen a game which does sound synthesis through physics simulation. But maybe this topic would be welcome on Sound Design Stackexchange. \$\endgroup\$ – Philipp Dec 1 '14 at 11:31
  • \$\begingroup\$ @MrKatSwordfish That's just attaching a prerecorded audio sample as a property to a material. Every 3d engine can do that. But what this question is about is completely procedural generation of sound by simulating soundwave generation and harmonics within a body. \$\endgroup\$ – Philipp Dec 3 '14 at 8:21
  • 1
    \$\begingroup\$ The Harmonic Fluids algorithm from two guys at Pixar works pretty well in auto generating water sounds. It may be a starting point if you're looking for procedurally generating sounds from scratch. youtube.com/watch?v=l95tZCl7YlQ \$\endgroup\$ – danijar Dec 3 '14 at 8:26
6
\$\begingroup\$

When you say 'synthesis' do you mean pure analog/additive/FM synthesis from scratch, or would a sample-based approach be acceptable? If you can't use combinations of real-world audio samples then this is more complicated process. Trying to generate truly realistic sounds through synthesis isn't the standard way that most game/virtual instruments/sound designers work.

If you do need something that's based on real waveform synthesis, like what you linked to, you probably need to perform a harmonic analysis of a large variety of sounds. Performing a harmonic analysis of various sounds will allow you to break real world sound samples down into fundamental sine waves. Keeping a data structure that associates each sound with its harmonics will probably allow you to interpolate new sounds semi-realistically. Using all this harmonic data, you can then construct the sounds again through additive synthesis. To do this from scratch you will probably need:

  • A large database of samples that you will associate with a wide variety of physical conditions. (In your case, samples of a variety of materials of various shapes at different masses.)

  • From there you will probably need to perform a harmonic analysis of each sample to break it into fundamental waves. Each sound can then be synthesized again via additive synthesis of its fundamentals.

  • Interpolating new sounds should then be as simple as comparing the fundamental waves of one or more sounds, and interpolating between them. Like light, the physics of sound is pretty complex, so the odds of generating a truly realistic sound this way are pretty slim in my opinion. But analyzing a larger, more diverse pool of samples will probably result in more realistic results.

If you can't use harmonic analysis of real world samples for some reason, the scope of this problem will become a lot bigger - as sound synthesis usually relies on combination/modulation of fundamental waveforms, you'll need to use physical modelling synthesis instead.


However, if sampling is acceptable, the question becomes how accurate of a simulation are you going for? From what I remember, Valve's Source engine has a relatively similar method of combining sound samples for the same/similar effect (when a plastic prop collides with a metal one, they generate a 'hybrid' sound). If that's acceptable, then you will need to:

  • Gather one or more audio samples for each material. The more specific your materials, the better (i.e. plastic sheet, hollow plastic, hollow metal, metal sheet, solid metal, thick wood, thin wood, etc.), and having multiple audio samples per material will create more variety and less computer-like, 'uncanny valley' repetition.

  • When a collision occurs you can mix the audio from the samples of both objects depending on the force of the collision, the mass of the objects, etc. More forceful collisions will probably result in louder sounds, while a more massive object will likely sound at a lower pitch than a less massive one. One again, if you can afford to record a larger pool of sounds for objects with different contours or masses it will likely increase the realism of your final result.

  • Finally, if you can create a 'physically-based' reverb effect based on your environment it will help ground your sounds in the environment. Things sound different in large rooms when compared to small rooms and outdoor environments.

I've never attempted this so I've only given it a little bit of thought, but from what I understand this is how most commercial virtual instruments create realistic sound performances. When 'VST' makers try to recreate the sound of a violin, they usually do so through extensive sampling of a real violin - sampling various notes on each string played at different velocities, using different playing techniques, and so on. Creating realistic sounds through pure waveform synthesis is interesting, but it's certainly not the standard practice in game engines at the moment. Good luck.

Edit: I didn't touch upon the physics engine part of your question, but I'd assume that most physics engines should be able to detect collisions for you and present you with information about those collisions (force/object mass/direction vectors/etc.).

\$\endgroup\$
2
\$\begingroup\$

You should check these for audio synthesis in C++. Check the sources as well. You may find a way to write audio buffer on the fly according to some parameters passed from physics engine.

\$\endgroup\$
0
\$\begingroup\$

From what I understand, you should be able to integrate the stream of haptic information from the CHAI3d or H3dApi frameworks with a DSP framework like http://aquila-dsp.org/.

The 3D haptics frameworks should provide the collision detection required, without adding the complexity of another physics layer. This will generate a force effect that can drive the DSP.

DSP frameworks give full control over sampled sounds using Fast Fourier Transforms (FFTs). This allows you to control:

  • Pitch
  • Doppler
  • Speed
  • Volume
  • Spatialization (3D stereo channels - look at Quaternions for the math).
  • Reverb
  • etc

The list goes on. There is some processing overhead with the FFTs, since they are doing a lot computations over samples that can be quite large.

There is a tradeoff between realism and processing. You may have to reduce the quality of your samples to get a reasonable effect.

Get a handle on how the outputs from the haptics will feed into the inputs of the DSP.

I have seen/experienced haptics automatically generated from sound streams in games, but the approach to drive the sound stream from the haptics would give a much more immersive realism.

Please post your progress, and good luck.

\$\endgroup\$

Your Answer

By clicking “Post Your Answer”, you agree to our terms of service, privacy policy and cookie policy

Not the answer you're looking for? Browse other questions tagged or ask your own question.