Author Topic: Binaural Audio (VR for the ears)  (Read 9026 times)

Offline Paul

  • Administrator
  • double
  • *****
  • Posts: 3499
  • Developer
    • View Profile
    • PaulsCode.Com
Binaural Audio (VR for the ears)
« on: September 05, 2011, 08:04:45 AM »
I was recently pointed to some pretty interesting technology.  A hearing aid company called Starkey Labs has developed an audio processing algorithm called Cetera, which mimics the way the ears pick up sound in the real world.  This stuff is pretty amazing - you can't even tell the sound is coming from the headphones (really sounds like it is right there in the room with you).  There is a nice example on this website.

The algorithm itself is not published anywhere (at least I couldn't find it), but I'm looking into "reinventing" it myself to include in my SoundSystem library as a new audio attenuation model, to allow for some awesome audio VR, and eventually even incorporate it into a new Mupen64Plus audio plug-in, to breath new life into those old Nintendo 64 games.

I'm looking at tackling the problem as a 3-step process:

1) Overall logarithmic attenuation based on distance from the listener, plus Doppler effect if enabled (the normal way of doing 3D audio, minus the panning)

2) Phase and attenuation adjustments per side based on the direction vector, using average real-world values for speed of sound, attenuation, and distance between the ears

3) Additional filtering per side to simulate high granularity multi-directional echoing (either by using a filtering formula derived from "audio ray tracing" a high-poly model of the ears and skull, or by doing the ray-tracing and mixing real time if it can be done quickly enough to avoid latency problems)
« Last Edit: September 05, 2011, 08:07:16 AM by Paul »
Device: Samsung Galaxy Nexus i515
CPU: TI OMAP4460, 1.2 GHz (dual core, ARM Cortex-A9)
GPU: PowerVR SGX540, 307 MHz
Resolution: 720 x 1280
Rom: omni-4.4.4-20141014-toro-FML KitKat 4.4.4, rooted

Device: Eee PC 1015PEM
CPU: Intel Atom N550, 1.5 GHz (dual core, x86)
GPU: Intel GMA 3150, 200 MHz (dual core)
Resolution: 1024 x 600
Rom: android-x86-4.3-20130725 Jelly Bean 4.3, rooted

Offline justcfx2u

  • bit
  • Posts: 1
    • View Profile
Re: Binaural Audio (VR for the ears)
« Reply #1 on: October 14, 2011, 08:29:38 AM »
Hey Paul,

From skimming over your list, you're definitely on the right track.  In my experimentation I strongly believe that early reflections are one of the most important aspects of direction-finding, so having a decent reverb engine (with EQ) is critical.  For some reason I want to also talk about decent comb filters too, especially for close-up stuff.

Anyhow, what you're looking for in technical terms (if you haven't already found it) is HRTF [Head-related transfer function].  That'll land you plenty of technical results, especially MIT KEMAR.  As a demonstration product, you could probably get your hands on WaveArts Panorama which uses HRTF in VST format so you have a working implementation to play with.

Instead of doing lots of hardcore math, you might be able to get away with using a series of "pre-cooked" Impulse Responses and run your inputs through them (using convolution).

The most unfortunate thing I've noticed with HRTF is that the different models which are out there are averages, approximations.  So the reference models which sound great to me likely sounds inaccurate to someone else, thus dampening the experience.

You may also find some interest in Ambisonics for positioning. Ambisonics has had some commercial success in Codemasters titles, though that is primarily for speaker-based systems, and not necessarily headphones (where all the best HRTF stuff happens).

As you are likely aware, most of these techniques work best only under optimal circumstances (speaker and listener placement, frequency response of output device, listening-environment crosstalk/cancellations, headphones which don't suck...) all play a huge role in the final experience.  Not to mention the ear's sensitivity for purpose of direction-finding is really good.  It all happens in the transient/initial stage of an event.  Which means, of course, it's really difficult to get right.

All this being said, I think 99% of all the (game) sound engines out there are terrible, and I would love to see some realtime simulation and even as little as 102% of the quality of existing solutions.  Of course I can't wait for audio ray tracing (and realtime ray-traced lighting!), done on a frame-by-frame basis (imagine Voxengo Impulse Modeler + convolution reverb), but this remains out of reach for now.  :)

Creative SoundBlaster folks got a decent idea to do reverb presets in hardware, especially during the later EAX models where a sound could transverse multiple 'layers' before reaching the 'player'.  While it never was all that awesome, it sure kicked the snot out of anything else out there at the time.  And it wasn't hard to use in level-creation too.  Level designers would define a rough 'sound area' whose radius encompassed your room or area, select a preset, and away you went.  Sure it was static, but it was still pretty cool.

But you know what I'd love even more?  Low-latency audio.  The buffers required for audio output is staggering (up to hundreds of milliseconds).  If we can push that to 10ms or less between what is visualized on-screen and what is heard, games get instantly more lifelike and immersive.  Think ASIO of WASAPI.

« Last Edit: October 14, 2011, 08:41:04 AM by justcfx2u »

Offline Benrox

  • bit
  • Posts: 9
    • View Profile
Re: Binaural Audio (VR for the ears)
« Reply #2 on: May 24, 2014, 04:02:28 PM »
lewl I just googled the audio thingy and dats some kewl sheet I didn't have headphones but my laptop has awesome sound anyway so it actually did sound good you can actually tell where the sound is but if you did actually remake this thingy you think its possible to add a converter to make something sound better? o-0