Developing for Oculus Rift and VR Platforms
- MikeQuell
- Site Admin
- Posts: 412
- Joined: June 6th, 2013, 4:15 pm
- Location: Los Angeles, CA
- Occupation: Sound Designer
- Contact:
Developing for Oculus Rift and VR Platforms
I was wondering if there were any Oculus (or VR) devs out there and if they could share their experiences with the hardware so far. I've heard people using "binaural plugins" to help increase immersion among others things. Having not had the opportunity to work with the hardware yet, I would love to hear more about it. What kind of problems have you ran into? What's the design process like and how thorough is playback testing in regards to referencing on different types of hardware (headphones)?
Re: Developing for Oculus Rift and VR Platforms
You mentioned hardware, but I've been playing with 3Dception. With any of the binaural plugins, they want a mono file to do their magic so that could complicate things if you also want to create a mix for people playing without headphones. It seems like a lot of these early VR games coming out are also playable as non-VR experiences so I imagine things will be easier when people aren't afraid to commit to a full VR game (because presumably they will use headphones only).
- joe_griffin
- Audio Specialist
- Posts: 78
- Joined: June 7th, 2013, 4:23 pm
- Location: Chicago
- Occupation: Recording engineer
- Location: Chicago
- Contact:
Re: Developing for Oculus Rift and VR Platforms
I've done one project for Oculus. It was pretty straightforward as far as prepping assets, but I also had a really fast turnaround schedule and almost no budget. So I wasn't able to do much with imaging, certainly not to the extent that I was using binaural sim plugs.
Since I work primarily in broadcast post, it was certainly a nice new challenge. I'm sure the video game vets here would roll their eyes at what I delivered...
I'd be interested to hear what other people are doing because I'm sure I'll be seeing more of these projects come up.
Since I work primarily in broadcast post, it was certainly a nice new challenge. I'm sure the video game vets here would roll their eyes at what I delivered...
![Embarrassed :oops:](./images/smilies/icon_redface.gif)
I'd be interested to hear what other people are doing because I'm sure I'll be seeing more of these projects come up.
Re: Developing for Oculus Rift and VR Platforms
Disclaimer: I lead the development of 3Dception, so this post is not a plug, but rather a summary of my experiences.
My first impressions with VR
Normal stereo panning over headphones doesn't do it. It's a bit bizarre to see a 3D world around you but have all the sounds playing inside your head. There's a huge disconnect when this happens. VR experiences aren't passive on a screen. You are inside a virtual world and it only feels natural that the audio should react correctly to it. This creates a lot of technical and creative opportunities, which I shall get to in a bit.
The scale of things becomes all the more important. Mixes done for a normal desktop screen do not feel right. I often have to rework the sense of scale and create a greater level of contrast between elements. HDR can be useful.
It quickly becomes apparent that audio will play a huge role in creating a convincing experience. More importantly, it allows us sound designers to let our work really shine.
Tech Considerations
Binaural audio is definitely the way forward. I've had some decent VR experiences in 5.1 (sitting on a swivel chair in the sweet spot), but the lack of elevation effects is apparent and you need a real good 5.1 system that is set up well.
Being able to mix for headphones is in some way liberating. Environmental noise, room acoustics and badly setup speakers will not destroy your carefully crafted mix
But binaural audio has different challenges and opportunities:
Opportunities:
![Smile :)](./images/smilies/icon_e_smile.gif)
I had mentioned some of this on DesigningSound.org many months ago.
My first impressions with VR
Normal stereo panning over headphones doesn't do it. It's a bit bizarre to see a 3D world around you but have all the sounds playing inside your head. There's a huge disconnect when this happens. VR experiences aren't passive on a screen. You are inside a virtual world and it only feels natural that the audio should react correctly to it. This creates a lot of technical and creative opportunities, which I shall get to in a bit.
The scale of things becomes all the more important. Mixes done for a normal desktop screen do not feel right. I often have to rework the sense of scale and create a greater level of contrast between elements. HDR can be useful.
It quickly becomes apparent that audio will play a huge role in creating a convincing experience. More importantly, it allows us sound designers to let our work really shine.
Tech Considerations
Binaural audio is definitely the way forward. I've had some decent VR experiences in 5.1 (sitting on a swivel chair in the sweet spot), but the lack of elevation effects is apparent and you need a real good 5.1 system that is set up well.
Being able to mix for headphones is in some way liberating. Environmental noise, room acoustics and badly setup speakers will not destroy your carefully crafted mix
![Smile :)](./images/smilies/icon_e_smile.gif)
Opportunities:
- Full 3D panning: it is great to hear sounds move above you, behind you and through every point in space
- The space around the player is another parameter to play with. It's not just about how loud a sound is in a mix, but also where it is relative to the player. Sounds that are noticeably behind the player will force him/her to turn around and localise it. Does the narrative/moment in the game call for such a reaction?
- Those distance attenuation curves need more work! (Yes, I see this as a good thing!)
- Detailed and good mixes have a lot of room to shine
- Headtracking tech has been the missing ingredient so far. Devices like the Rift do a great job at capturing even minor head movements, which are very important in resolving localisation confusion
- Stereo/multi-channel files are not ideal. (In 3Dception we've been adding features to make this easier)
- Mixing ambient audio is new territory. Some of the sound designers I know use a mix of stereo and binaural sounds for ambience. The stereo sounds help create a diffused sound field, while the binaural sounds fool the brain into believing that everything is properly spatialised
- Pre-recorded binaural audio can be useful, although, not so much when the player's position and orientation changes drastically
- A good number of the VR games use the environment around the player or props in the scene to integrate UI elements. Audio can play a crucial role here in drawing the player's attention to something that might not be in their line of sight
- Too much of binaural panning can be confusing (remember those early stereo and surround mixes?). HDR mixing and some culling can help
- Translating binaural mixes to non-headphone setups can be challenging. Even if headphones are a norm, everyone likes it if mixes are compatible (this is another area we're working on to simplify)
- Binaural audio is not just about panning, but also the environment. Environmental effects are crucial. Reverb wet/dry ratios have never been more important (another area we're spending a lot of time on)
![Smile :)](./images/smilies/icon_e_smile.gif)
I had mentioned some of this on DesigningSound.org many months ago.
- MikeQuell
- Site Admin
- Posts: 412
- Joined: June 6th, 2013, 4:15 pm
- Location: Los Angeles, CA
- Occupation: Sound Designer
- Contact:
Re: Developing for Oculus Rift and VR Platforms
Great stuff, Varun! What has your experience been like with 7.1 headphones, or ones that simulate surround sound. I would think that would help fill in the gap for a game with binaural modeling missing.
I'm a big fan of using mono files as much as I can in standard/console development. For me, it just seems to make the mix a lot cleaner and also creates a hell of a lot more movement in the space. I have to imagine, like you said, it's even more significant under a VR scenario. I'll bookmark your site and check it out. I'm pretty ignorant on how the modeling process is done and very curious about it. Thanks for sharing your experiences so far!
I'm a big fan of using mono files as much as I can in standard/console development. For me, it just seems to make the mix a lot cleaner and also creates a hell of a lot more movement in the space. I have to imagine, like you said, it's even more significant under a VR scenario. I'll bookmark your site and check it out. I'm pretty ignorant on how the modeling process is done and very curious about it. Thanks for sharing your experiences so far!
Re: Developing for Oculus Rift and VR Platforms
I've done a few projects now using the Unity game engine and Oculus Dk2. Really glad to see a thread go up. I'm dying for insight on the subject from other designers working on Oculus stuff. So far my observations are ..
- 1st Person - Both projects were a '1st person experience' so all my observations are from that perspective. Working on a racing game for example I'm sure many of these observations would be less useful.
- New Expectations - I totally agree with Varun, what you might expect to work using your desktop monitor and speakers doesn't translate at all when you 'strap-on' the headset and headphones. It becomes immediately obvious that your regular approach to setting up 3D sound in a game engine won't cut it at all. Your expectation about what the sound is supposed to do is much more demanding.
- Oops its not nailed down yet - Getting the development environment setup for Oculus and Unity wasn't super easy. Certainly there are piles of un documented things to discover just getting your development environment setup. Take your time and do your research. Plan some extra time to tech the hardware you'll be using. For example getting the game window to fill the Oculus headset wasn't an 'it just works' thing it took a lot of dorking around.
- It's 3D not 2D stupid I'm sure most (sound designers) assumed this from the get go but I figured I just put it out there. I've seen a few bare bones executions (just a video projected into 3D space) where the sound was just stereo. It doesn't work. You can't get away with slapping music in stereo and VO down the centre and walk away. People expect the sound 'to be 3D' (whatever that means) and that something is going to 'happen' with the sound when the turn their head around (This is based on actual comments people made comparing 2D stereo audio and 3D in engine stuff) Do the project a favour and plan for something to specifically directly address sounds panning around in 3D space. Sounds obvious but I've seen projects (done in game engines) that didn't.
- Laggy Sound Engine - The limitations of the Unity audio system were obvious as soon as I started moving my head around I could feel / hear the calculation and attenuation of 3D audio roll-offs lagging far behind, this is consistent with desktop projects I've worked on but somehow exaggerated with the Oculus. I had to actually design around this a bit for one of the projects where you needed to turn your head quickly as part of the gameplay / experience.
- Cone it if you got it - Early on in production common feedback I received on the sound was 'it doesn't feel 3D enough' and I totally agreed. I wish I could have had Varuns 3Deception implemented in time there just wasn't the time to experiment. My simple yet effective solution was to make the 3D roll off max about x10 smaller than I normally would have and setting up some custom curves. What I ended up with was small sound bubbles around props and attached to anims' so that in combination with pushing the Unity audio listener out from the player a few Units was the cheapest solution to a listener 'cone' . I believe something like this has been implemented in Wwise recently. By cone I mean right now the average listener is an omni directional thing but if you're able to reduce the listener 'angle' to something like a cone (imagine a spotlight shooting straight out from the player) I think that will help solve this 'it isn't 3D enough' problem. I'm looking forward to an upcoming project this year and experimenting with this new Wwise listener angle / cone solution
-
- Intern
- Posts: 1
- Joined: January 16th, 2015, 2:24 pm
- Location: College Park, Md
- Occupation: Software Developer
Re: Developing for Oculus Rift and VR Platforms
Note: I'm the lead developer on VisiSonics/RealSpace3D Audio plugin. So, no plugs here either, just observations and thoughts.
My observations/experiencing from using the Oculus hardware and integration w/ Unity is very simple. The only problems I found are what you find with anything that interfaces with open hardware and that is integration problems with different graphics card, and if some machines have DSP cards. My biggest obstacle was getting the Rift to work in a multiple monitor setting but, after scouring the web I was able to find the solution that worked for my machine environment.
I'm highly susceptible to motion sickness so I feel a little woozy after using the DK1 & DK2. So, when doing a lot of testing w/ either DK1 or DK2 I hold the HMD in my hand, moving it around, and viewing the mirrored monitor. I must say Crescent Bay fixes that problem for me.
The audio in Unity is 3D panning where you get left/right when turning your head in the direction of the source but, above, front and behind stays stuck in the middle of your head.
I second what Varun has already said.
RealSpace3D is a HRTF engine to produce binaural sound. HRTF engines in the past just were not that good so today when some folks hear the term they cringe, shrug, or get down right angry. I think a couple of companies have shown (ours and Varun's company being another) that HRTF is getting a better reputation.
So, that being said somethings to consider when designing your audio for a good VR experience is to not overdue your scene with 3D audio. Noise is noise...3D or not. Having a lot of objects emitting 3D audio is going to make the game listener not be able to localize the sounds properly or miss out on an audio cue that you may want them to really notice.
3D audio engines that provide environmental reflection calculations means you need to be aware of how you wish your audio to sound reflecting off of those materials.
Lastly, do not design your audio in a bubble. VR is truly new so get as many people experiencing how things sound in your VR world, and gaining feedback as to whether or not it is believable and sounds good.
That's my input and experiences.
My observations/experiencing from using the Oculus hardware and integration w/ Unity is very simple. The only problems I found are what you find with anything that interfaces with open hardware and that is integration problems with different graphics card, and if some machines have DSP cards. My biggest obstacle was getting the Rift to work in a multiple monitor setting but, after scouring the web I was able to find the solution that worked for my machine environment.
I'm highly susceptible to motion sickness so I feel a little woozy after using the DK1 & DK2. So, when doing a lot of testing w/ either DK1 or DK2 I hold the HMD in my hand, moving it around, and viewing the mirrored monitor. I must say Crescent Bay fixes that problem for me.
The audio in Unity is 3D panning where you get left/right when turning your head in the direction of the source but, above, front and behind stays stuck in the middle of your head.
I second what Varun has already said.
RealSpace3D is a HRTF engine to produce binaural sound. HRTF engines in the past just were not that good so today when some folks hear the term they cringe, shrug, or get down right angry. I think a couple of companies have shown (ours and Varun's company being another) that HRTF is getting a better reputation.
So, that being said somethings to consider when designing your audio for a good VR experience is to not overdue your scene with 3D audio. Noise is noise...3D or not. Having a lot of objects emitting 3D audio is going to make the game listener not be able to localize the sounds properly or miss out on an audio cue that you may want them to really notice.
3D audio engines that provide environmental reflection calculations means you need to be aware of how you wish your audio to sound reflecting off of those materials.
Lastly, do not design your audio in a bubble. VR is truly new so get as many people experiencing how things sound in your VR world, and gaining feedback as to whether or not it is believable and sounds good.
That's my input and experiences.
- joe_griffin
- Audio Specialist
- Posts: 78
- Joined: June 7th, 2013, 4:23 pm
- Location: Chicago
- Occupation: Recording engineer
- Location: Chicago
- Contact:
Re: Developing for Oculus Rift and VR Platforms
Hey y'all. I know this thread is a couple of months old, but I'm starting to get more requests for VR/3D sound and I wanted to ask more about it. The project I spoke about earlier in this thread was one where I just prepared a bunch of assets and handed them off to a programmer, so for all intents and purposes I'm a bit of a newb.
I'm trying to get my head around the various software requirements and I'm still a bit lost. Can these sorts of designs be done within ProTools (I ask because that's my main platform right now) or do I need to invest in Unity/3Dception etc.? I'm totally willing to go to my bosses with a shopping list, but I'm looking for a little guidance as far as putting it together.
I also wonder about using binaural source recordings. How would such a recording work within the VR environment? Would it "Stay still" as the user turns his/her head?
Thanks all.
I'm trying to get my head around the various software requirements and I'm still a bit lost. Can these sorts of designs be done within ProTools (I ask because that's my main platform right now) or do I need to invest in Unity/3Dception etc.? I'm totally willing to go to my bosses with a shopping list, but I'm looking for a little guidance as far as putting it together.
I also wonder about using binaural source recordings. How would such a recording work within the VR environment? Would it "Stay still" as the user turns his/her head?
Thanks all.
Re: Developing for Oculus Rift and VR Platforms
Hey Joejoe_griffin wrote:Hey y'all. I know this thread is a couple of months old, but I'm starting to get more requests for VR/3D sound and I wanted to ask more about it. The project I spoke about earlier in this thread was one where I just prepared a bunch of assets and handed them off to a programmer, so for all intents and purposes I'm a bit of a newb.
I'm trying to get my head around the various software requirements and I'm still a bit lost. Can these sorts of designs be done within ProTools (I ask because that's my main platform right now) or do I need to invest in Unity/3Dception etc.? I'm totally willing to go to my bosses with a shopping list, but I'm looking for a little guidance as far as putting it together.
I also wonder about using binaural source recordings. How would such a recording work within the VR environment? Would it "Stay still" as the user turns his/her head?
Thanks all.
So your question about what's required is entirely project specific. Passing assets to a programmer is about the worst workflow possible. Feel free to contact me directly if you still have questions about how to get setup. It's not really a shopping list of equipment you need rather a set of skills and experience required but there is a small gear side re: an oculus headset, game engine licenses, audio middleware etc .. . I've just completed my 3rd Oculus based project and moving on to a fourth in the spring.
- joe_griffin
- Audio Specialist
- Posts: 78
- Joined: June 7th, 2013, 4:23 pm
- Location: Chicago
- Occupation: Recording engineer
- Location: Chicago
- Contact:
Re: Developing for Oculus Rift and VR Platforms
Excellent, thanks! I'll try to get my thoughts together and send you a message.