does ogre support virtual reality development ?

[fla666]

I have a project of virtual reality will be initiated and now begin to choose proper engines , i think the cadidates may contain unreal , unity3d ,ogre , or other commercial 3d engines ,

could somebody tell me if ogre can support VR development conveniently ? if it support , does ogre have some advantage comprared to other engines ? thanks !

[xrgo]

In my company we do virtual reality using Ogre, its possible but it need a bit more work than other engines (unity?).
We are using Oculus DK2, but we are doing lens distortion by shader... that means no time warping, and no warning dialog (yay?!)... but doing lens distortion by sdk is possible in ogre, but for some reason we had very low fps with that approach so we stick to shaders =)
We are using Ogre because we love Linux, open source stuffs, c++, many external libraries, having control over everything! and paying no royalties . We are now creating an engine from scratch using Ogre 2.1 and its looking good in graphics and performance

[cybereality]

You can do VR with OGRE, but if you are targeting Oculus then you will have an easier time with Unity or Unreal as they include official support.

Early on (with DK1) I actually put together a VR app in OGRE (just some floating cubes as a test) and it only took about a week or two. However, DK2 has added some complexity that make custom integrations more difficult.

Kojack was actually working on getting DK2 support working in OGRE, but I'm not sure what the status of that is now.

One thing I will say: don't bother with pixel shader based distortion. You can get close to something working, but it will likely be subtly wrong and make people uncomfortable. Ideally you should let the Oculus SDK handle the distortion for you.

Hope that helps.

[tmason]

You can do VR with OGRE, but if you are targeting Oculus than you will have an easier time with Unity or Unreal as they include official support.

Early on (with DK1) I actually put together a VR app in OGRE (just some floating cubes as a test) and it only took about a week or two. However, DK2 has added some complexity that will make custom integrations more difficult.

Kojack was actually working on getting DK2 support working in OGRE, but I'm not sure what the status of that is now.

One thing I will say: don't bother with pixel shader based distortion. You can get close to something working, but it will likely be subtly wrong and make people uncomfortable. Ideally you should let the Oculus SDK handle the distortion for you.

Hope that helps.

I'd love to see what Kojack finalizes with in terms of VR with Ogre3D. Is there a codebase available for the process made currently?

[spacegaier]

Kojack was actually working on getting DK2 support working in OGRE, but I'm not sure what the status of that is now.

If you wait a few more days, chances are quite high, that there will be an update in that regards .

[Kojack]

I'm working on it at the moment, I was planning on getting it going soon.
But the oculus sdk has just wasted a day of my time (and a lot of mental momentum) with odd behaviour.

I've got a working test program that does opengl and directx 11 on the DK2 with sdk 0.4.4.
I made a new app for my cleaned up version and started porting the code over. But the new one has the following behaviour:
- headset won't turn on (stays in standby mode)
- can't do debug runs in directx 11, it hits memory access errors when creating the DirectX11 device that don't happen in non debug runs.
- gl3plus results in flickering to black, even with a really basic compositor that just clears the screen to a solid colour.

I've spent the day rearranging code and copy/pasting between the app that works and the one that doesn't.
Finally, at around 8pm, I found the problem. It seems the entire oculus direct mode rendering is dependent on the ovrHmd_GetEyePoses function.

ovrHmd_GetEyePoses gives you the orientation and position of each eye (based on the headset sensors). I wasn't calling it because I'm not doing head tracking in the demo yet, I just wanted rendering to work first. But it's ovrHmd_GetEyePoses that seems to actually do everything! The rift only turns on when that one function is called. It also stops opengl from flickering (seems that without it the oculus render shim that injects into opengl does double buffer swaps or something).
Why does ovrHmd_GetEyePoses turn on the rift instead of ovrHmd_AttachToWindow or ovrHmd_ConfigureTracking? That would be like OIS corrupting Ogre's rendering if you don't read the mouse coordinates at least once.

Anyway, now I can get back to work.
Things are a little tricky though. Parts of the Oculus sdk need to be run before Ogre starts so they can hijack the library loading code to take over the system directx and gl libs. Parts need to be run after ogre starts. Plus special stuff needs to be done in between (setting up compositors, multiple cameras, etc). So there's a three stage manual init process so far (you need to call initPreOgre(), initOgre() and initPostOgre() at the right times. Although I might be able to merge the last two, I haven't thought about it much yet).

AMD's new Liquid VR has a feature called Direct To Display that seems to be the equivalent of Oculus' Direct mode. If we can get Nvidia to do the same, maybe Oculus won't need to hack around with system libraries to trick windows into not seeing the hmd as a monitor. (Every VR headset needs a Direct mode equivalent, extended mode sucks in comparison).

Ok, enough complaining, back to code...

(At the moment it's targeting Ogre 2.0, that's my main dev environment at the moment. It might work in 2.1 too, I'll test it at some point. Ogre 1.x might be possible at a later time, the compositor stuff is too different between them so I need to write it twice)

[Kojack]

I should point out that's just Oculus stuff I'm working on at the moment. The top post was just for VR, not specifically oculus.
For that, Ogre 2.1 already comes with a stereo view demo, any non-distorting (so probably narrow fov) VR headset with side by side 3d rendering would work with that right now, for example Vuzix headsets. It would just be a matter of setting the fov and res correctly then using an sdk to read the headset orientation.
Oculus headsets require special support because of their custom distortion rendering, windows bypassing direct mode and stuff like timewarp.

So for general VR, it may already be possible with Ogre 2.1. For oculus rift vr, it's on the way.


I'd actually like to support multiple headsets. Any companies want to send me a free one? CastAR, Hololens, Vive, OSVR...

(No, seriously, I want a CastAR, but I'm not buying one until the sdk is public. So far it seems only people who bought the headset get the sdk. I didn't buy the rift until I checked the sdk out, I had a working ogre demo before even ordering)

[Klaim]

Damn, so many VR possibilites now, I would be rich I would have sent you one of each XD

[tmason]

I'm working on it at the moment, I was planning on getting it going soon.
But the oculus sdk has just wasted a day of my time (and a lot of mental momentum) with odd behaviour.

I've got a working test program that does opengl and directx 11 on the DK2 with sdk 0.4.4.
I made a new app for my cleaned up version and started porting the code over. But the new one has the following behaviour:
- headset won't turn on (stays in standby mode)
- can't do debug runs in directx 11, it hits memory access errors when creating the DirectX11 device that don't happen in non debug runs.
- gl3plus results in flickering to black, even with a really basic compositor that just clears the screen to a solid colour.

I've spent the day rearranging code and copy/pasting between the app that works and the one that doesn't.
Finally, at around 8pm, I found the problem. It seems the entire oculus direct mode rendering is dependent on the ovrHmd_GetEyePoses function.

ovrHmd_GetEyePoses gives you the orientation and position of each eye (based on the headset sensors). I wasn't calling it because I'm not doing head tracking in the demo yet, I just wanted rendering to work first. But it's ovrHmd_GetEyePoses that seems to actually do everything! The rift only turns on when that one function is called. It also stops opengl from flickering (seems that without it the oculus render shim that injects into opengl does double buffer swaps or something).
Why does ovrHmd_GetEyePoses turn on the rift instead of ovrHmd_AttachToWindow or ovrHmd_ConfigureTracking? That would be like OIS corrupting Ogre's rendering if you don't read the mouse coordinates at least once.

Anyway, now I can get back to work.
Things are a little tricky though. Parts of the Oculus sdk need to be run before Ogre starts so they can hijack the library loading code to take over the system directx and gl libs. Parts need to be run after ogre starts. Plus special stuff needs to be done in between (setting up compositors, multiple cameras, etc). So there's a three stage manual init process so far (you need to call initPreOgre(), initOgre() and initPostOgre() at the right times. Although I might be able to merge the last two, I haven't thought about it much yet).

AMD's new Liquid VR has a feature called Direct To Display that seems to be the equivalent of Oculus' Direct mode. If we can get Nvidia to do the same, maybe Oculus won't need to hack around with system libraries to trick windows into not seeing the hmd as a monitor. (Every VR headset needs a Direct mode equivalent, extended mode sucks in comparison).

Ok, enough complaining, back to code...

(At the moment it's targeting Ogre 2.0, that's my main dev environment at the moment. It might work in 2.1 too, I'll test it at some point. Ogre 1.x might be possible at a later time, the compositor stuff is too different between them so I need to write it twice)

This is great progress. Sucks that the Oculus SDK is hard; I could only get Extended Mode working myself with a wide range of SDKs.

Glad that you are trying to do this in a way that will work for a range of VR headsets and not just Oculus.

[duststorm]

I'd actually like to support multiple headsets. Any companies want to send me a free one? CastAR, Hololens, Vive, OSVR...

I agree that the right way to support these devices in Ogre would be through OSVR. Let the work to get a device working be done once in an opensource implementation, and then reuse it endlessly.
The more momentum the OSVR project gets, the more headset (and other VR device) vendors will be forced to support it. Everyone wins.
Kojack, check the osvr website, they are offering companies and developers the chance to join. If you get accepted you will get an OSVR headset kit for free when they launch in june. I think it's worth dropping them a line and telling them you are working on integration with the Ogre3d engine.

[Kojack]

I'll check it out.

One issue with Ogre and Oculus is that in direct mode you must have a monitor of at least 1920x1080. If you have a 1680x1050 like me, you can't render at full res. This is because you need to make a window on a normal monitor for the oculus sdk to take over in direct mode, but you can't make a window bigger than the monitor. In directx 11 it's possible to make the swap chain bigger than the window (the oculus sdk takes over the swap chain) so you could have a 10x10 window with a 1920x1080 swap chain, that works fine and is the kind of thing we want, but Ogre doesn't support it, I need to hack the source code. I don't know yet if opengl supports that.

At the moment I have it running (missing stuff), but I'm getting double images whenever I move my head. That really needs to be worked out.

I'm also going to have to hook into the compositor some how (I don't know the new compositor well), I need to call time warp code between two compositors (one for stereo vision, one for distortion).
So there's at least 5 places in an ogre program that need to call parts of the oculus sdk.

[Kojack]

The double image thing is strange. I thought it was because of low persistence not being active (a similar effect happens in the oculus demo world when low persistence is disabled). But I added a key to toggle it, low persistence was definitely on. Without it the screen brightens and there's motion blur.
It's also not a problem with camera poses being out of sync between the two eyes. If I close one eye I still see two images overlapped.
It's not a problem with my head movement or the hmd tracking, I bypassed that and put in a hard coded camera rotation, the problem is still there.
Screenshots, video capture and the monitor don't show it. It's like the oculus sdk is averaging two frames together when it does the direct mode stuff, or it's doing a double buffer swap. The official demos aren't showing it.

[cybereality]

What kind of framerates are you getting? Because with the DK2 if you get lower than the refresh (75Hz) it can appear as a sort of double-image.

[Kojack]

1000+fps with vsync off, 75fps with vsync on.
(It's just a ninja, sinbad, a few lines and a quad).

[Kojack]

Some more testing with the sdk demo world, it seems the problem is that overdrive isn't turning on. If I disable overdrive in the demo it has the same effect.

Looking through the sdk source, it seems client mode (where we do everything) doesn't support overdrive, that distortion cap flag is only checked in sdk rendering mode. In fact all distortion cap flags (timewarp, chromatic aberration correction, etc) passed to ovrHmd_CreateDistortionMesh are ignored (with the comment: "Not used now, but Chromatic flag or others could possibly be checked for in the future.")

So looks like I'm either going to have to work out how overdrive is performed and recreate it or dump client rendering and switch to sdk rendering.
For those that haven't heard of it, client rendering is where we do all the rendering within ogre. SDK rendering is where we render to a texture, then the oculus sdk takes over and does all the distortion and stuff and handles the rest of the rendering using it's own mini 3d engine. While sdk rendering is easier, I really don't like the idea of two different rendering pipelines mixing together, especially when the least stable one is partly closed source. I don't trust the sdk to not mess up ogre's rendering state (it already breaks ogre's dx9 and f***s up ogre's opengl if you aren't careful, and that's just the first init call). CLient rendering also gives us more control, such as using custom distortion shaders. Guess I don't have a choice though.

So... a bit of reverse engineering on overdrive mode, then probably back to the drawing board if that fails...

Or just watch tv, I need to get up for work in 8 hours.

[tmason]

So looks like I'm either going to have to work out how overdrive is performed and recreate it or dump client rendering and switch to sdk rendering.
For those that haven't heard of it, client rendering is where we do all the rendering within ogre. SDK rendering is where we render to a texture, then the oculus sdk takes over and does all the distortion and stuff and handles the rest of the rendering using it's own mini 3d engine. While sdk rendering is easier, I really don't like the idea of two different rendering pipelines mixing together, especially when the least stable one is partly closed source. I don't trust the sdk to not mess up ogre's rendering state (it already breaks ogre's dx9 and f***s up ogre's opengl if you aren't careful, and that's just the first init call). CLient rendering also gives us more control, such as using custom distortion shaders. Guess I don't have a choice though.

My recommendation is SDK rendering; you get the benefits of Oculus's research and updates while also shifting much of the legal responsibility to them should someone want to sue a person using Ogre3D for eye damage while using an Oculus Rift.

Always use the manufacturer's core code \ SDK if possible.

[Kojack]

Always use the manufacturer's core code \ SDK if possible.

Client rendering is a feature of the oculus sdk, it just gives far more control to the user. But the docs missed some of the features, the shader they gave for client rendering is lacking features hidden in the sdk rendering version.
My concern is that since the sdk had broken opengl support for so long, how will it interact with Ogre 2.1's AZDO pipeline? Maybe it won't have a problem. It just seems icky to have one 3d engine pass textures to another engine every frame.

[tmason]

Always use the manufacturer's core code \ SDK if possible.

Client rendering is a feature of the oculus sdk, it just gives far more control to the user. But the docs missed some of the features, the shader they gave for client rendering is lacking features hidden in the sdk rendering version.
My concern is that since the sdk had broken opengl support for so long, how will it interact with Ogre 2.1's AZDO pipeline? Maybe it won't have a problem. It just seems icky to have one 3d engine pass textures to another engine every frame.

I see; I am certainly no Ogre3D expert but all I can say is that giving control of the distortion to the manufacturer for their products is a win.

Just seems like the right thing to do; given that they have the research (medical and 3D graphics team wise) behind the shaders you are getting with their SDK. Plus their improvements will be incorporated into the pipeline.

[Kojack]

Plus their improvements will be incorporated into the pipeline.

Kind of.
Client rendering uses a shader from the oculus sdk doc. (They give us the shader in the docs and the sdk has all the calls to set up the distortion mesh)
SDK rendering uses a shader that's hard coded into the sdk's static library.

This means that in fact any improvements to the pipeline won't affect existing games using SDK rendering, they need to be recompiled to get things like an updated shader. Whereas client rendering can update the shader at any time without recompiling, it's just a normal file on disk along with the rest of ogre's shaders.

In either case, the mesh distortion is the same. It's just do we draw it or do they.


Urgh, I'm sitting at work with my dk2, trying to get it working in Unity 5 for a class. I'd rather be at home using it in Ogre.

[tmason]

Plus their improvements will be incorporated into the pipeline.

Kind of.
Client rendering uses a shader from the oculus sdk doc. (They give us the shader in the docs and the sdk has all the calls to set up the distortion mesh)
SDK rendering uses a shader that's hard coded into the sdk's static library.

This means that in fact any improvements to the pipeline won't affect existing games using SDK rendering, they need to be recompiled to get things like an updated shader. Whereas client rendering can update the shader at any time without recompiling, it's just a normal file on disk along with the rest of ogre's shaders.

In either case, the mesh distortion is the same. It's just do we draw it or do they.

Hmmm, but don't you get a lot of other benefits with using the SDK (timewarp, predictive tracking, etc.)?

Or will you be incorporating that into your work as well?

And please know that I appreciate all of your time on this. Having proper VR incorporated into Ogre3D will be a massive benefit to all!

[Kojack]

All of that is available in client too. It's just not automatic.
(It's surprising how much things have changed since DK1, there's a lot of stuff going on now)

I'll try sdk rendering soon. (probably friday, work's a bit busy at the moment)


( I can imagine cybereality is sitting at his desk in the oculus office yelling at the monitor "just use sdk rendering!". )

[tmason]

All of that is available in client too. It's just not automatic.
(It's surprising how much things have changed since DK1, there's a lot of stuff going on now)

I'll try sdk rendering soon. (probably friday, work's a bit busy at the moment)

OK, Thank you for all that you do. I'll shut up for now since I am not the brightest when it comes to Ogre3D and VR.

Just some thoughts that I had

[cybereality]

( I can imagine cybereality is sitting at his desk in the oculus office yelling at the monitor "just use sdk rendering!". )

Yeah, I was thinking to say "just use SDK rendering" but I'm actually *not* in the office right now. If you can use it, SDK rendering is the better bet.

[Kojack]

All of that is available in client too. It's just not automatic.
(It's surprising how much things have changed since DK1, there's a lot of stuff going on now)

I'll try sdk rendering soon. (probably friday, work's a bit busy at the moment)

OK, Thank you for all that you do. I'll shut up for now since I am not the brightest when it comes to Ogre3D and VR.

Just some thoughts that I had

Hehe, no prob.

Argh, I wish my new laptop would arrive at work. Running unreal engine 4 on my old one with the dk2 gives horrible framerate for a room with a few boxes.


One benefit of client rendering: we could do the improved text rendering without affecting game rendering performance technique that oculus discussed at gdc but haven't released yet. That required customising the distortion shader.

I also want to try the Valve stencil mesh method of speeding up vr rendering, but that will work with either style (it happens in the initial stereo render stage in ogre, not part of the distortion).

[Kojack]

Looks like SDK rendering in DirectX11 is out for now. The oculus sdk needs access to the swap chain, but the ogre dx11 render system keeps that as a protected member with no accessors. I can get the back buffer via getCustomAttribute and the device by casting the render system pointer then calling dx11 render system specific members, but mpSwapChain is hidden inside of D3D11RenderWindowSwapChainBased.

Does anybody know of a way to get a IDXGISwapChain pointer from a dx11 device or back buffer?

I guess it's going to be opengl only for now. Back to work...