Oculus Rift in Ogre

[Kojack]

SDK is now live.

Here's a quick test based on the shader from the sdk docs. I don't have all the values right yet (I need to query the oculus library for the 4 distortion coefficients, right now I'm just guessing and hard coding).



That should (once the values are set right) work on an Oculus.
There's no head tracking in there yet.

Next steps:
- sleep (it's 7:20am, time for bed)
- fix the shader and camera values to be correct instead of guesses
- add head tracking
- rewrite the whole thing, since I'm using my work framework which I can't give out.

While the visual side is easy to test even without an oculus, the head tracking can only be tested by someone with the actual headset. Luckily the dev kits have already started arriving, at least two users have had it delivered, so somebody else can test it. I haven't ordered yet, I just want ogre to get in early.

[hedphelym]

sweet! Thanks for doing this.
I'll post in here as soon as it arrives, I can test.

[Kojack]

Here's some initial shader stuff I'm using. Bits are copy/pasted from the oculus sdk (mainly just the HmdWarp function), other bits I wrote.

oculus.cg

Code: Select all

sampler2D RT : register(s0);
uniform float2 LensCenter;
uniform float2 ScreenCenter;
uniform float2 Scale;
uniform float2 ScaleIn;
uniform float4 HmdWarpParam;

// Scales input texture coordinates for distortion.
float2 HmdWarp(float2 in01)
{
	float2 theta = (in01 - LensCenter) * ScaleIn; // Scales to [-1, 1]
	float rSq = theta.x * theta.x + theta.y * theta.y;
	float2 rvector = theta * (HmdWarpParam.x + HmdWarpParam.y * rSq + HmdWarpParam.z * rSq * rSq + HmdWarpParam.w * rSq * rSq * rSq);
	return LensCenter + Scale * rvector;
}

float4 main_fp(float4 pos : POSITION, float2 iTexCoord : TEXCOORD0) : COLOR
{
	float2 tc = HmdWarp(iTexCoord);
	return float4(tex2D(RT, tc).rgb,1);
}

oculus.material

Code: Select all

fragment_program Ogre/Compositor/OculusFP_cg cg
{
	source oculus.cg
	entry_point main_fp
	profiles ps_4_0 ps_2_0 arbfp1
	
	default_params
	{
		param_named LensCenter float2 0.5 0.5
		param_named ScreenCenter float2 0 0
		param_named Scale float2 0.1 0.1
		param_named ScaleIn float2 2 2
		param_named HmdWarpParam float4 1.5 1 1 1
	}
}

material Ogre/Compositor/Oculus
{
	technique
	{
		pass
		{
			depth_check off

			vertex_program_ref Ogre/Compositor/StdQuad_vp
			{
			}

			fragment_program_ref Ogre/Compositor/OculusFP_cg
			{
			}

			texture_unit RT
			{
				tex_coord_set 0
				tex_address_mode border
				tex_border_colour 0 0 0
				filtering linear linear linear
			}
		}
	}
}

oculus.compositor

Code: Select all

compositor Oculus
{
    technique
    {
        texture rt0 2048 2048 PF_R8G8B8

        target rt0 { input previous }

        target_output
        {
            // Start with clear output
            input none

            pass render_quad
            {
                material Ogre/Compositor/Oculus
                input 0 rt0
            }
        }
    }
}

In the compositor you may notice that it's using a 2048x2048 render target. That's just a quick test hack. The oculus rift is 1280x800 (resulting in 640x800 per eye). But due to the barrel distortion shader magnifying the centre of each eye, you need higher res to avoid blockiness. The sdk says how to calculate the required dimensions. Something like 1600x1000 should be enough apparently (I haven't tested yet).

[Kojack]

It's a bit hard to test without actual head tracking, so I added the tracking code for my Vuzix 1200VR headset.
While I'm not overly happy with the headset itself, the Vuzix sdk is very nice. Just call IWROpenTracker() to turn it on, then call IWRGetTracking() every frame to read the current pitch, roll, yaw values. There's calibration functions too, very easy to use.

Hmm, maybe I should make this demo be Vuzix and Oculus compatible. Both use side by side stereo and have 3dof tracking, I just need to turn off the barrel compositor for the vuzix. That way I can run it here too.
Now to take a look in the Oculus tracking sdk.

(In case any Oculus devs are hear, don't freak out. I'm not going to break the eula rule against using the oculus sdk code with another brand of headset. The vuzix stuff will be separate)

Mini rant: I hate it when companies release an sdk but require you to specify what project you intend to use the sdk for. With the Oculus Rift, you have to name your project before you can get the sdk, others require project descriptions.
Well, some of us don't know exactly what we want to make yet! We want to learn what the sdk can do before deciding on a project! We want to do some R&D to find potential uses. Requiring us to know in advance what our one specific use of the sdk will be is stupid. I had this problem with photoshop years ago. I applied for the sdk (it wasn't public back around photoshop 5). They wanted to know why I wanted it (no, shut up and give it to me!). I told them for filter development and adding my own file format import/export support (I had a 96bit tiled streaming file format for my raytracer). They responded that the sdk didn't support adding file formats, I had to pay for the premium sdk to get that. I would have known that if they gave me the damn sdk first so I could see what it exposed!
For the latest Vuzix sdk I need to give them my phone number and a description of how I intend to use the sdk before I can download it! Luckily a hard drive search found a copy of their old sdk that I got back in 2011.
Ok, end of rant.

[Kojack]

The main demo that comes with the Oculus Rift is a house with a garden and surrounding distant town.
The art assets for the demo are standard textures and a 14MB xml file describing all the geometry. I've just added a rapidxml loader to my test so it can load the same level and textures. This way I can compare the distortion a bit better.

Official Oculus demo:


Ogre's version:


As you can see there's a few differences. My one is more distorted (not surprising, I'm guessing at the values in the shader) and it's darker. The demo uses baked lightmaps, I'm guessing that maybe the official one treats them as srgb or something. I haven't looked into what their lighting setup is (they might be using a custom lighting shader or hdr or something).

I'm not recreating the physics from the demo, I don't feel like that right now. Maybe later.

[Kojack]

New ogre version, this time using some of the Oculus sdk values:



Now that's a lot closer to the real thing.

New parameters for the shader:

Code: Select all

	default_params
	{
		param_named LensCenter float2 0.5 0.5
		param_named ScreenCenter float2 0 0
		param_named Scale float2 0.3 0.35
		param_named ScaleIn float2 2 2
		param_named HmdWarpParam float4 1.0f, 0.22f, 0.24f 0
	}

[Kojack]

Ok, lighting makes sense now. The official demo uses this in it's lighting shader:

Code: Select all

color2.rgb = color2.rgb * lerp(1.2, 1.9, saturate(length(color2.rgb)));

where color2 is the lightmap. I'm using fixed function, so mine is just the original color2.


A margarita and some shader coding later and here's the ogre result. Groovy, now it looks right.

[Wolfmanfx]

Awesome!

[Kojack]

I like that it requires zero changes to ogre to get this stuff running, just make two cameras, two viewports, stick both cameras at the right spacing on a scene node and turn on a compositor. Then you control the scene node just like in any other ogre game, but it's rendering stereo.

I also just ordered an oculus, so in a few months I may be able to see my demo running in person.

[Klaim]

This is pure awesomeness.

[druidsbane]

I've had similar code for my virtual desktop project for a while now. Same idea: http://hwahba.com/ibex. I haven't used the coefficients from Oculus but now that my Rift is here I probably should give it a shot looking forward to seeing if this gets integrated, especially the sensor code or what your plans are! Good job

[Kojack]

I really shouldn't be writing a demo using a framework I can't give out, so I've scrapped the demo and started from scratch using just the standard ogre sdk.

[Kojack]

I've just re-added Vuzix headset support to the new demo so I can test the camera controls while a head tracker is active. Not too bad. I've got fps style mouse aim, but with the headtracker then applied afterwards. Movement is based on body orientation (the mouse), so while walking forwards you can look left or right. It's like trackir mode in Arma 2.
But I think I need a crosshair to indicate body direction.
In a game with a cockpit (car, space sim, plane, etc) it's a lot clearer which way you are facing.

Hehe, imagine Trespasser with an Oculus and a wiimote or Hydra. Not only can you look at yourself, but one arm has pretty much full control and performs physics on the world.

I've got pretty much everything implemented now, in theory a real oculus should be able to control the demo. I still need to add back in the camera settings from the sdk, some error messages and more controls.
I also converted the tuscany map (from the sdk) into an ogre mesh file. I was lazy, the entire map is one mesh with a ton of submeshes (probably over a hundred), but that was easier because the original xml file has all vertices in world coordinates. Anyway, I want a better level, this is just for testing.

The oculus code is currently is a class I'll put on the wiki at some point. Pretty small. It sets up the oculus (enables tracker and stuff), sets up ogre (makes the viewports and cameras) and can give you the head orientation.

Binary demo hopefully coming soon.

(Hmm, I keep bashing the 1200VR, but now that I've got it working with full head tracking in ogre, it actually looks cool. The tracking is laggy (like 0.3 seconds) and the screens don't look good, but it's still cool. Other games like HalfLife 2 didn't work with it as well, because they don't support head rolling. Using a 3dof tracker on a 2dof game is less immersive than laggy tracking)

Oh yeah, I'm going to add xbox 360 input too.

[Kojack]

Binary time.

First up, this is just a testing demo to see if the headset tracking and distortion are working on the real device. I removed the tuscany scene, it looks cool but even zipped it took up over 60MB. So this one is just the ogre sibenik mesh (the building from the ssao samples) and the sinbad model.
I'll get something much better looking (and better controls) later.

OgreOculus V0.2 (7MB)
http://www.mediafire.com/?avunf95hf78yk37 Edit: Added 2 april 2012

Controls:
wasd - movement
1 - walking mode (clamped to the ground)
2 - flying mode
r - reset pitch of your body
escape - exit
mouse - look around

There's two orientations in the demo, your body and your head. Body is controlled by the mouse. Head is controlled by the oculus rift (if it's detected). They combine together for the actual view you see. Movement is relative to your body orientation only, so looking around with just your head won't change the direction you are moving.

Both OpenGL and DirectX9 are available in this demo. Either will work, but I'd recommend DirectX9 because in Ogre it gives you the ability to specify the monitor to use, while OpenGL mode starts where ever it wants to.
The monitor is chosen (assuming you are using DirectX9) in the Ogre config dialog that appears when you start the demo. Look for the "Rendering Device" option. It gives you a list of display outputs. One of them should be your Oculus Rift. You can also set vsync on or off here (on is recommended) and change the screen res (1280x800 is the native Oculus res of the devkit).

If anything goes wrong (mainly oculus based, I'm not trying to debug ogre itself right now), please post (or send me) your ogre.log file.

[ophiucus]

Kojack, this looks amazing ! very impressive work.

Do you consider sharing the source code ? to see how you managed the integration ?

[Klaim]

Do you consider sharing the source code ? to see how you managed the integration ?

The oculus code is currently is a class I'll put on the wiki at some point. Pretty small. It sets up the oculus (enables tracker and stuff), sets up ogre (makes the viewports and cameras) and can give you the head orientation.

[Kojack]

Yep, source will be released once somebody confirms it actually works.
(I've got this on the official oculus forums too, not just here)

[Kojack]

At the moment this style of stereo rendering is fairly simplistic. It works fine, but it's not as fast as possible. There's two cameras, two viewports and two compositors running. This means:
- double the scene processing (each camera culls separately)
- double the geometry processing (objects are vertex shaded twice per pass, once for each eye)
- similar pixel processing (same number of pixels are drawn regardless of mono or stereo)

With DX10 level hardware and api (probably opengl compatible, definitely not dx9) we can get away with one camera, one viewport.
We make two render textures. (A 2 element texture array)
Only one camera and viewport is used.
The original way had each camera doing it's own frustum culling against the scene. But the two frustums have identical top, bottom, near and far planes. Only the left and right planes vary, and then only by around 6cm. So instead we make the camera's frustum slightly wider. Now it covers what can be seen by both cameras (not much difference, so slight false positives shouldn't hurt much). Now the scene is only processed once.
A geometry shader is needed in all materials. The shader takes in one triangle and duplicates it, moving the results to simulate stereo cameras. Render target array indexing is used to send the original and the copy to different render textures (one per eye). Now we have vertex processing only happening once (good for things like animations), with final view and projection happening in the geometry shader to the two output triangles.
The two compositor instances (one per viewport) are replaced with one full screen compositor that takes the two render textures and concatenates them into one screen. (At least I think that will work)

As a final touch, deferred lighting could be used. While the g buffer stuff would be per eye, the actual lighting can be applied full screen to the results of the concatenation.

You could also do something with shadows. Texture shadows won't be affected (much) by stereo eyes, so shadows could be rendered once then used for each eye (with offsetting to match the cloned geometry position).

At least that's what I came up with in the shower this morning. I don't really feel like implementing this stuff though. Plus I do all my using ogre stuff in dx9. It's just an idea in case framerates drop.

[TheSHEEEP]

I'm gone for a longer weekend to wander around on a mountain.
I come back, and awesome things have happened.

I should leave more often

[mkultra333]

Great stuff Kojack.

For a while I used an experimental 3DVision mode in my project that I made up, that let me use all my standard shaders. I have since modified things to just use the normal 3DVision stereo rendering, but the old code is still there so I'm thinking it might be ideal for adding O.R. support later.

Like you mention above, I just made a slightly wider frustum so I only had to do culling once. I use one camera, I shift it back and forth between the two different eye positions and render the scene to two different textures, which then get combined at the end onto a single texture. If I then process that final texture with the distorting shaders, will that kind of thing work with the Oculus? I'm assuming that in the end, it just cares about some texture you pass it from the graphics cards, and is indifferent to how you created that texture.

[Kojack]

Yay, apparently the head tracking works fine, but the distortion isn't. The oculus doesn't position the lenses in the centre of each screen half, I need to offset it.

If I then process that final texture with the distorting shaders, will that kind of thing work with the Oculus? I'm assuming that in the end, it just cares about some texture you pass it from the graphics cards, and is indifferent to how you created that texture.

Yep, that should be fine. The shader is designed for one eye only (you can't use it as is to distort a full screen), but that shouldn't be too hard to change.

[Kojack]

New release version 0.2:
http://www.mediafire.com/?avunf95hf78yk37

I've added in the lens centre calculations. It looks like it might be right, but I'll need somebody to test for me again.
If the value isn't right, you can now press 3 and 4 to move the value around. It writes the current value to the ogre.log as you change it (so if it's wrong, let me know what's in the log).

I also disabled camera pitch control using the mouse (vertical movement) if a headset is detected. You can still yaw with the mouse at any time (with or without headset).

[madmarx]

That's really great! As always Kojack, you are the best of us .

out of context :
For those interested in multiples rendering from a few close cameras, I read that the rendering is also possible through a "segment camera" : render into 1 'long' texture all the informations, and then draw both quad using that texture. But I wouldn't recommand it since it requires to calculate some elements on CPU (though it could be doable on GPU since opengl4.3 with the fragment counters), appart if you need to draw 8+ cameras. The paper is "The Epipolar Occlusion Camera" by Paupescu.

[Kojack]

No, I'm just quick to jump on new things then just as quick to get bored and move on.

Here's a new release, hopefully it's working better now (turns out I was only doing offset stuff in the view matrix and shader, but it was needed in the projection matrix too).

Version 0.3: http://www.mediafire.com/?8f2cjp9r2bk22yj

[Kojack]

I really shouldn't make releases at 5:30am after not sleeping.
I had added back in the tuscany level for testing the appearance against the official demo, but then changed back to the sibenik mesh (the one in the pic above) but forgot to remove the line that loads the tuscany resources. Since they aren't in the 7z file (it would make the release 10 times bigger, and I don't know the redistribution rights for them) it failed to load.

Fixed now, version 0.4: http://www.mediafire.com/download.php?ayr5h2316mhb64t