[2.1] Bug with prepass and MSAA (Dx11)

[al2950]

Hi

I have been chasing a nasty bug that I cannot solve, which is caused by enabling MSAA (FSAA). This issue can be clearly seen in the SSR sample but it may appear in other areas more subtly. The main issue is that with the texture binding goes to crap, so if you add a diffuse texture in the SSR sample it is obvious. I have created a branch if anyone wants to help!
https://bitbucket.org/al2950/ogre-2.x-u ... sMsaaIssue

Please see below for rendering without and with MSAA, and also a renderdoc screen shot.

SSR without MSAA

SSR with MSAA

Renderdoc screen shot

As you can see in renderdoc there are a number of missing textures. That draw call is a scene_render with use_prepass. But there are other passes missing textures as well.

Any help/advice would be appreciated! I have not tested in GL as GL does not support MSAA and MRT at the moment.

[dark_sylinc]

Fixed. Thanks for the report.
Yeah, it was just two missing lines, causing the diffuse textures to override the wrong slots.

[al2950]

Fixed. Thanks for the report.
Yeah, it was just two missing lines, causing the diffuse textures to override the wrong slots.

God damn it, seems so simple now. Thank you very much, that was driving me mad!

[rujialiu]

Fixed. Thanks for the report.
Yeah, it was just two missing lines, causing the diffuse textures to override the wrong slots.

God damn it, seems so simple now. Thank you very much, that was driving me mad!

Great! I also had this problem but didn't have time to investigate. Now it works for me, too. Thank you for reporting!

However, when MSAA is on, I'm unable to resize the window with the following error:

Code: Select all

17:03:25: OGRE EXCEPTION(3:RenderingAPIException): D3D11 device cannot Clear State

Error Description:ID3D11DeviceContext::VSSetShaderResources: Resource being set to VS shader resource slot 5 is still bound on output! Forcing to NULL.

ID3D11DeviceContext::PSSetShaderResources: Resource being set to PS shader resource slot 5 is still bound on output! Forcing to NULL.

 in D3D11RenderSystem::_setRenderTarget at C:\ogremygui\OGRE\RenderSystems\Direct3D11\src\OgreD3D11RenderSystem.cpp (line 2179)

Do you have this problem?

[al2950]

In short no! I checked my engine, as well as the base ogre samples from the Ogre version I am using.

HOWEVER, I am seeing some other nasty issues. I am getting random NANs that appear in a single channel of a texture that result in bad things! Also I still have 'missing' slots, in the case of my original post the gbuf_depthTexture is still missing, which I suspect maybe be causing some issues, but I doubt it would be the cause of your issue. I have not had time to investigate yet as I had a deadline to integrate the Vive, so I have fallen back to a standard Forward renderer instead of a hybrid one.

FYI, i am using Ogre version e3106edf58f8034076e48b507f21e383b00c2fd8, which I suspect you are to!

**EDIT** Your error does ring some bells, but I cant remember what. I will let you know if I remember!

[dark_sylinc]

The resizing error probably means we're using a depth buffer that no longer exists (dangling pointer) and if so, very bad things can happen afterwards without crashing. I will try to fix this today (which would explain al2950's artifacts).

[al2950]

The resizing error probably means we're using a depth buffer that no longer exists (dangling pointer) and if so, very bad things can happen afterwards without crashing. I will try to fix this today (which would explain al2950's artifacts).

Any chance you managed to make any progress with this. I have debugged it and as far as I can tell it is add the texture correctly to the command buffer etc, at least from Ogre's point of view. I have not checked the directX texture its self....

[al2950]

I came back to this issue today I found the problem(s) but dont know the best fix.

@rujialiu
I am pretty sure I found your issue although I can not prove it as I dont get the crash. Basically in 'HlmsPbs::postCommandBufferExecution' it unbinds the MsaaDepthTexture but it unbinds the wrong slot.
It is doing the following;

Code: Select all

size_t texUnit = mGridBuffer ? 1 : 3;

but it should be

Code: Select all

size_t texUnit = mGridBuffer ? 3 : 1;

@dark_sylinc
The reason the depth texture is missing is because it is using the same depth pool. So its trying to use it as a depth texture as well as an input texture. Whats the best solution here? Simply copying into another texture?

[dark_sylinc]

I came back to this issue today I found the problem(s) but dont know the best fix.

@rujialiu
I am pretty sure I found your issue although I can not prove it as I dont get the crash. Basically in 'HlmsPbs::postCommandBufferExecution' it unbinds the MsaaDepthTexture but it unbinds the wrong slot.
It is doing the following;

Code: Select all

size_t texUnit = mGridBuffer ? 1 : 3;

but it should be

Code: Select all

size_t texUnit = mGridBuffer ? 3 : 1;

Thanks! Fix pushed.

@dark_sylinc
The reason the depth texture is missing is because it is using the same depth pool. So its trying to use it as a depth texture as well as an input texture. Whats the best solution here? Simply copying into another texture?

You'll have to keep digging I'm afraid. Since DX11it is valid to use a depth buffer as both RTT and texture as long as all depth writes are off (in other words it is read only).
D3D11DepthBuffer::createReadOnlySRV should be getting called which creates an SRV (Shader Resource View) of the Depth Buffer for binding as RenderTarget with flag D3D11_DSV_READ_ONLY_DEPTH, which is what prevents D3D11 from complaining the depth buffer is bound as both target and texture; and when binding, getDepthStencilView should be getting called with VP_RTT_READ_ONLY_DEPTH set, so that mDepthStencilView[1] is used (and not mDepthStencilView[0] which is not-read only).

[dark_sylinc]

I re-read the thread... after fixing the unbinding in postCommandBufferExecution, what bugs remain? The NaN thing?

Could you post pictures?

[al2950]

hmmm interesting. I still have the NAN issue, sometimes there is no issue, move the camera then NANs appear, sometimes in a single channel sometimes in all channels.
Here is what the scene should look like;

This is what it looks like when it goes 'wrong'. NB the white out is caused by HDR.

Again when looking at renderdoc the depth buffer remain unbound during the scene render as shown in the first post. The depth buffer appears to be set correctly ie read only

Any further ideas!? Ill debug the stencil details you mentioned

[dark_sylinc]

Are you using the latest RenderDoc version? Because there was a RenderDoc bug where it incorrectly treated a DepthBuffer w/ read only access + used as a texture to be unbound from texture (in fact it was our sample the one that discovered it ).

A workaround at the time was to make all depth buffers use stencil (use PF_D32_FLOAT_X24_S8_UINT instead of PF_D32_FLOAT) so RenderDoc would treat it correctly. Perhaps this bug wasn't fixed correctly or you're in an old version.

As for the NaN... I suspect what's going on here could be something else. Can I have access to a few RenderDoc captures? (good and bad).
My hunch is that the problem could be in how we approach MSAA. You see, to get correct MSAA we need SV_Coverage with the values after depth test. But for some reason SV_Coverage returns the values before depth values. You can read me rant about it in a blogpost. Otherwise we end up sampling the wrong values when two objects overlap (and you're having NaNs at the borders...).

The solution is that we manually go through each subsample and use an heuristic to get one subsample.
Well, it should be always right if it weren't for floating point precision errors. In theory we should search for the subsample whose depth matches exactly our depth. In practice we chose any subsample that looks like its depth is approximately similar to the depth we're rendering.
We get it right most of the time, but sometimes it can be wrong, and this is what could be causing your NaN.
Something that I haven't yet tried is, instead of this approximation (which has precision errors, turning the algorithm into more of an heuristic) we could try picking the subsample with the lowest negative ulp difference; and perhaps that could/should guarantee us to always pick the right subsample.

Anyway, in case you're lost, I have two main theories:

• When we pick the wrong subsample, it picks an invalid normal or roughness value which ends up causing a NaN. And because you're using HDR, this NaN spreads to the next frames as well. What value you clear the GBuffer to could make a tremendous difference (particularly avoid normals set to 0 0 0).
• There's a driver bug. Combine MSAA, floating point bit reinterpretation, depth buffers bound as both target & texture, branches, SV_Coverage, firstbitlow all in one single place and you have the perfect cocktail to wreak havoc in a driver. I wouldn't be surprised if it works OK on another GPU.

This is why I'd want a closer look at your case. The NaN could be a byproduct of picking the wrong subsample, or I may not be able to repro it at all because it's a driver bug.

[al2950]

Ha, you were 100% spot on, I was using am old version of RenderDoc! I had just assumed it was infallible!

I have to admit I took a look at your MSAA code and your blog post and it left me feeling a little bit, well, stupid!

Anyway have created 2 renderDoc captures as requested;
https://file.town/uploaded/kao7xl9lylbzx4q7oyfj2pclq

Please note even the version that is 'working' has some aliasing issues so the MSAA code is not working as expected. its not so obvious in the capture by I get bright or black swimming artefacts on the edges. I have tried it on 2 graphics cards, a 780 and a 980, same issue..

[dark_sylinc]

I took a look at your captures. I'm not sure what exactly is going on; and it took me a while to find the NaNs.

SSR uses the image rendered last frame as source for the reflections for this frame (reprojected, to account for camera motion). You're definitely carrying over the NaNs from previous frame, which is the reason you can't get rid of the problem once it starts. The real challenge is that it's hard to capture w/ RenderDoc the exact frame where the NaN was produced.

If you were able to make a deterministic run where the bug appears (timeSinceLastFrame would always be a fixed value), you could be able to capture the Nth frame where the bug appears.

What we're seeing now is just a left over from the bug.

I'm still researching though. The assembly of one of the shaders is clearly doing a div instruction, and if we ever do 0 / 0, it's game over.

One thing to note though: You clear the GBuffers to (1, 1, 1) which produce an invalid normal (not unit length). Try to use a better clear colour like 0, 0, 1 (at best you're wasting performance by forcing SSR raytrace unnecessarily, at worst you're causing a bug). Edit: Ouch, the sample does this too.

Edit: Ok, looking at Samples\Media\2.0\scripts\materials\ScreenSpaceReflections\HLSL\ScreenSpaceReflectionsCombine_ps.hlsl we perform a lot of risky operations:

• isoscelesTriangleInRadius performs a division. I would have to check the math, but it may end up doing 0 / 0
• Even if isoscelesTriangleInRadius doesn't do 0 / 0; it might still return 0 (or a value close to 0); which later on become a NaN when we perform log2( incircleSize * p_depthBufferRes.w )

If I'm reading this right, technically neither of these values should affect the generated colour itself because they're not mixed together, but these values end up being used in sampling instructions, and who knows how the GPU will react if we pass NaN texture coordinates or mip levels.

Since you're the one who can repro the problem, try aggressively sanitizing all these inputs and see if the problem persists. If it disappears, then start removing the sensitization until the culprit(s) are revealed.

Edit 2:
Alternatively, newColor may have become infinite for some reason (over exposure?), and then:

Code: Select all

newColor.xyz *= ( 1.0f - abs(remainingAlpha) );

If remainingAlpha = 1; then it becomes infinite * 0; which results in NaN colour.

Edit 3:
Does this happen with 4xMSAA? SSR + MSAA 8x might not work the same.

[al2950]

I took a look at your captures. I'm not sure what exactly is going on; and it took me a while to find the NaNs.

Sorry about that, I am completely forgot to point you towards the rough draw call

I have tried all your suggestions and the problem still persisted, although it was still prominent in MSAA x4, it was much less prominent in MSAA x2.

If you were able to make a deterministic run where the bug appears (timeSinceLastFrame would always be a fixed value), you could be able to capture the Nth frame where the bug appears.

Well after many many many many attempts I finally managed to find my illusive NaN! The NaN is actually created when rendering the ocean mesh, but it only appears in one of the sub samples of the MSAA texture, in this case sub sample 6. It appears at the very top of the ocean mesh.

Anyway the good news is that when I stop rendering the ocean the problem goes away, so Ogre is fine. The bad news I have no idea how to go about trying to fix this, so if you have any tips please let me know! Its not actually my code but I do have the shader source. Not sure what may cause a single sub-sample to render as a NaN.....(Infact its a single channel of a single pixel of a single sub sample!)

Thanks very much for your help on this!

[al2950]

OMG RenderDoc is so awesome! Being able to actually debug pixels line by line is incredible!

Now I have a very interesting issue if any would like to help, which seems that the rasteriser is going mad. Basically the vertex shader outputs a value between 0 and 1, however the rasteriser for a certain MSAA sub-pixel (actually set of sub pixel), outputs values above 1 some as high as 9.* which eventually causes a NaN in the pixel shader. Is this a bug in the rasteriser or some weird expected behaviour. If anyone is interested I can point them to the exact pixel and shader registers in a render doc dump....

Please let someone be interested in this!!

[dark_sylinc]

Out of curiosity I would like to see a capture of that (but please highlight Event ID and the pixel coordinates and msaa subsample where that happens)

[al2950]

Yay!

Please find RenderDoc capture here;
https://file.town/uploaded/qjsze9w1c0rjgnqn6617aiqv2

The NaN pixel can be found in Draw call EID: 777, pixel: 374x265, MSAA sample 6. Please note that even though only that pixel causes a NaN, all pixels on the horizontal line 265 are incorrect for MSAA sample 6.

If you were to debug that pixel the offending shader lines are 229-234

Code: Select all

mov r0.xyz, fogColor.xyzx
add r1.xyzw, -r0.xyzw, r1.xyzw
mad r0.xyzw, v2.zzzz, r1.xyzw, r0.xyzw
log r0.xyz, r0.xyzx
mul r0.xyz, r0.xyzx, oneOverGamma.xxxx
exp o0.xyz, r0.xyzx

Which corresponds to the code

Code: Select all

float4 finalColor = lerp(fogColor4, waterColor, fogFactor);
finalColor.xyz = pow(finalColor.xyz, float3(oneOverGamma, oneOverGamma, oneOverGamma));

The NaN is caused because fogFactor (v2.zzzz) is 9.48077, but its meant to be between 0 and 1. This creates a negative number outputted from the lerp, and Pow of a negative number is a NaN.

The interesting bit is of you look at the mesh output from the vertex shader, the fogFactor (V2.zzzz) which corrosponds to the FOG semantic (gets optimised into TEXCOORD 2) is always between 0 and 1, mostly 0.9* to 1.0. So I have no idea how such large values are being created in the rasteriser stage.

Enjoy! I have only tested this on a Nvidia machine. So looking for an AMD GPU to see if it does something similar.