[2.2] Instanced stereo questions. Unlit, quad passes, etc

[zxz]

Hello!

I am trying out the instanced stereo support in 2.2 (GL3Plus RS). For the most part it seems to be working well, but I have found a couple of issues.

1. The Unlit shaders seem to be missing support for instanced stereo. Am I correct in this assumption? I suppose that only some smaller shader modifications are missing, although I am somewhat lost in all the macro expansion stuff going on.

2. Is there some nice way of handling instancing together with quad passes, for example for sky rendering? As an example, can a setup similar to TutorialSky_Postprocess be adapted to handle both instanced and non-instanced rendering?

Thanks

[zxz]

2. Is there some nice way of handling instancing together with quad passes, for example for sky rendering? As an example, can a setup similar to TutorialSky_Postprocess be adapted to handle both instanced and non-instanced rendering?

Hmm. I suppose that the quad pass must be adjusted to draw two quads with the camera directions of each individual eye's frustum corners. Ogre::Camera doesn't seem to provide corners for both eyes, so some code needs to be added to compute those using the eye projection matrices.

[dark_sylinc]

Hi!

We had issues trying to get instanced stereo to work with "sky as a postprocess" technique (there's the clipping against the middle of the screen, we also had aspect ratio issues which cause dizziness when you tilt your head and due to the head's movement nature this is impossible to prevent), which is why we implemented SceneManager::setSky( true, SceneManager::SkyCubemap, texture ); method.

[zxz]

Hi!

SceneManager::setSky seems to use the same technique for drawing the sky, and the sky is only drawn in the left eye viewport when it is used. From what I can see there is nothing in its implementation that takes stereo rendering into consideration. Perhaps I am misunderstanding you?

I don't understand what the problems you describe are. If each eye were to be rendered individually with each eye's individual camera direction vectors, things should look good. That should be achievable if quad passes are aware of the camera's VrData. That should give the same result as when rendering the whole scene twice (once for each eye), which looks good. Is there something that prevents that from working?

[dark_sylinc]

SceneManager::setSky seems to use the same technique for drawing the sky, and the sky is only drawn in the left eye viewport when it is used. From what I can see there is nothing in its implementation that takes stereo rendering into consideration.

Uh oh, this must be Unlit's missing support for stereo instancing. Should be easy for me to fix.

I don't understand what the problems you describe are. If each eye were to be rendered individually with each eye's individual camera direction vectors, things should look good. That should be achievable if quad passes are aware of the camera's VrData. That should give the same result as when rendering the whole scene twice (once for each eye), which looks good. Is there something that prevents that from working?

I thought the same too, but it was tricky to get the math right. We could've made a mistake in the formulas though.

[zxz]

Uh oh, this must be Unlit's missing support for stereo instancing. Should be easy for me to fix.

The sky implementation uses old materials with custom shaders, not unlit, so I don't think they are related. In any case, I don't think the instancing the same primitive will take care of the eye differences without some additional mechanism making use of the individual projection matrices. I suppose it would work if an actual skycube was used though.

I am also very interested in instanced stereo support for Unlit though, as we cannot begin using instanced stereo until that works. I was looking into implementing it for a bit, and saw that Unlit differs somewhat from Pbs. It has support for identity world transforms, and uses complete worldViewProj matrices where Pbs uses only viewProj, and thus passes somewhat different data to the shader. So it's not quite as trivial as a copy/paste which was my initial hope.

I too think that it would be easy for you to fix . Now I don't mean to preempt your plans, but it would be very useful to know if that is something that might be fixed in the near future, or if I should go ahead and attempt to hack together some partial solution. These two things (sky and unlit) are what's blocking us from fully functioning VR at the moment. Anyway, it is really nice to (almost) be able to just flip a switch and get stereo rendering without having to setup multiple compositors and managing multiple cameras manually. Just some details missing.

Best regards!

[dark_sylinc]

I took a look and I forgot Unlit was never ported to shared code (i.e. most code written in .any files). I'm beginning to do that now.

[zxz]

Nice! How is it coming along? I suppose that the instanced stereo support comes along for the ride when the shared code is used?

[dark_sylinc]

I've pushed the "unification" changes to its own branch (to prevent accidentally breaking everyone until it's more tested). I'm done for now (for now I'm limiting it to vertex shaders, since it's all we need for instanced stereo changes).

I've used up the time I had left to focus on this issue for this weekend , so I will be doing the instanced stereo changes during next weekend.

Without the unification, the changes would've needed to be done 3x times.

[dark_sylinc]

I just pushed Instanced Stereo support for HlmsUnlit to its own branch.

I could not test it thoroughly since I don't have the use cases like you do (also I don't have the VR headset anymore, they were borrowed); but I tried to simulate a basic one and it seemed to be working as intended.

If you find no issues I'll be merging it to master.

[zxz]

Hello!

I have briefly tested the branch and it seems to work well so far! I have tested using a HMD where it looks good. It will get some more testing as we start testing VR more later.

[dark_sylinc]

Ahhh nice!

This is a low risk change (last famous words) so I'm going to merge it into master

[zxz]

Hi!

We had issues trying to get instanced stereo to work with "sky as a postprocess" technique (there's the clipping against the middle of the screen, we also had aspect ratio issues which cause dizziness when you tilt your head and due to the head's movement nature this is impossible to prevent), which is why we implemented SceneManager::setSky( true, SceneManager::SkyCubemap, texture ); method.

Hi!

I might be revisiting VR soon, and this problem was the main reason that made instanced stereo unworkable for us last year. The problem being that quad passes didn't get the correct camera direction vectors when instanced stereo rendering is enabled.

Do you know if any progress has been made on this front since last year? A quick look at the current quad pass code suggests that the situation might not have changed since then. If not, do you have any suggestions regarding how it could be resolved?

Thanks!

[zxz]

I don't understand what the problems you describe are. If each eye were to be rendered individually with each eye's individual camera direction vectors, things should look good. That should be achievable if quad passes are aware of the camera's VrData. That should give the same result as when rendering the whole scene twice (once for each eye), which looks good. Is there something that prevents that from working?

I thought the same too, but it was tricky to get the math right. We could've made a mistake in the formulas though.

Well, I looked into this once again, and the solution turned out to be as easy as I had thought at first. Somehow I managed to mess up the details when implementing it before. The trick to get the proper view vector for an asymmetric eye frustum (they often are in VR applications) is to preserve the length differences between the frustum corner vectors before interpolation across the quad. Then the interpolation works properly, and the vector can be normalized in the fragment shader (or used directly for cubemap lookup). Ogre might already be getting this right for the single quad drawn by a PASS_QUAD with CAMERA_DIRECTION. Done this way, the sky aligns perfectly at infinity and there is no longer any left/right offset causing double-vision.

I've created a custom pass to implement this, but the proper solution would be to make PASS_QUAD stereo-aware, then stereo rendering would Just Work for users. I haven't looked into making the changes inside of Ogre.

PASS_QUAD should draw a geometry that spans both eyes with normals calculated by each eye's individual projection matrix (Camera::getVrProjectionMatrix(i)). Of course, two quads can be drawn instead, but that is a bit more wasteful. I suppose that the same thing could be done for the other mFrustumCorners-alternatives, but I haven't looked into those cases. There doesn't seem to be anything complicated to take into account.

[Hilarius86]

I am also in the process of migrating from 2.1 to 2.2 and after using the instanced stereo branch I am now at reactiving skyboxes.
Did you come back to looking into the changes in Ogre or are you able to share your custom pass, so I can have look to adapt it for me?
I am also doing some other post processing and need to make that instanced_stereo aware as well.