- Try
- Rate
- Share
How did you rate this?
The Inside Story
Earlier this year, 麻豆社 R&D teamed up with the Soundscapes for Wellbeing project and 麻豆社 Archive to investigate how listening to natural soundscapes can improve our mental health and connection with nature. In this pilot, we wanted to explore whether binaural audio and augmented reality technology can improve how we interact with soundscapes, and what design challenges there might be.
Our sound designers worked with wintry poems read by Mary Anne Hobbs (Radio 6 Music) and Nick Bright (Radio 1Xtra). We also made a few soundscapes reminiscent of warmer climes including a tropical beach soundscape produced by Sian Eleri (Radio 1).
The app works best on your computer or Android device. You may experience audio glitches on iOS, particularly in the more complex scenes.
How it was made
We started off exploring augmented reality (AR) — visualising sounds as 3D objects drawn over a camera view of the listener’s real world surroundings. To prototype quickly, we used existing open-source software, including Google’s Resonance library for dynamic spatial audio rendering, and the AFrame library for describing scenes of 3D objects and working with AR devices. However, we met some stumbling blocks along the way, where the audio rendering performance was reduced by the processing power required to update the AR graphics. In the end, we opted to use a static 360-degree photo as a background, and only use the device’s 3-DOF orientation sensors instead of the camera. You can read more about our prototypes and findings from a co-design workshops on the R&D blog, and we hope we’ll be able to fully explore the 6-DOF AR option in the future.
For the sound design, we used the “mixer mode” tool to collect and listen to the sound effects. It includes access to a database of over 33,000 sound effects, and we could create simple soundscapes as demonstrated by Radio 1 Relax’s Sian Eleri as part of . The mixes were then imported into our Soundscape Explorer app, where they are spatialised by a binaural renderer. Binaural audio gives the illusion of listening to a real space by processing each sound source based on its distance and direction relative to the listener’s ears. By rotating the view or grabbing and repositioning the sounds, we can tune the mix to be just right. And you can do it too: the mixer tool makes it easy for anyone to create a soundscape, and you can share it to the app to explore it in 3D space. We found that it works really well even with just five to ten sounds, so just give it a go; we can’t wait to hear what you make.
Find out more
Using the sound effects mixer and positioning sound sources in a virtual space is much like the work of professional sound designers — using slightly different tools, such as our EAR production suite, and often working with composers and their own recordings, of course.
If you’re into binaural sound, we have some examples in this .
If you’re inspired to take your mixes further, why not check out some of the recordings from our recent Sounds Amazing event, including a talk on the soundscapes of 麻豆社 drama Vigil, and an insight from producer Benbrick into the detail of his work.
