[2.1] ESM Shadow Bug in GLSL PBS?

[Kaylx]

Hey peeps,

After trying to track down why ESM shadows aren't working well for me I came across what appears to be a bug in the GLSL shaders.

In HLSL and MSL there is a 'float2 padding' variable after ShadowReceiverData.shadowDepthRange but in GLSL it's missing. Indeed even the C++ code expects padding. Fixing the shader appears to make things worse so I don't know what's going on.

Kaylx

[dark_sylinc]

The padding isn't there because of how alignment rules work in GLSL vs HLSL & Metal.

It would help a lot to know a lot more detail to help you:

• Screenshot of the bug
• If the bug also happens with PCF. Screenshot with PCF for comparison
• Check the debug content of the shadow maps (as shown in ShadowMapDebugging sample)
• OS and GPU involved
• Whether if the issue only happens with GL, or also D3D/Metal (unless you're on Linux obviously)
• Ideally a way for us to reproduce the issue, such as compositor setup, lights, camera settings, objects in it. If you can repro the issue by modifying the ShadowMapDebugging sample, that would be superb

Remember for ESM you should call hlmsManager->setShadowMappingUseBackFaces( false ); if you aren't doing that already.

[Kaylx]

Sorry i didn't provide much detail before. I wasn't at the computer I was on the train posting from my phone.

Ok so i have a Windows 10 (64-bit) laptop with dual graphics: Intel HD 530 and NVIDIA GeForce GTX 960M.
PCF shadows don't work at all for me on Intel. I get the following errors:

Code: Select all

OpenGL:error(high) 1282: Error has been generated. GL error GL_INVALID_OPERATION in BindFramebuffer: (ID: 3854903065) Generic error
OpenGL:error(high) 1282: Error has been generated. GL error GL_INVALID_OPERATION in FramebufferTexture: (ID: 2521807278) Generic error
...
 GLSL validation result : 
Validation Error: Samplers of different types point to the same texture unit

They do work on NVIDIA and the issue i have with ESM does not exist.

The ESM issue happens on Intel and NVIDIA using OpenGL and DirectX.
I managed to reproduce in the ShadowMapDebugging sample and have attached the modified ShadowMapDebuggingGameState.cpp.
A one line change to the ground plane breaks the shadows. If i can call setCastShadows(false) i get the big black area.

I tried to debug the shaders and i may be on to a red herring but it looks like fDepth inside ShadowMapping_piece_ps.any could be causing the problem and that traces back to outVs.posL@n.z in ShadowMapping_piece_vs.any. I'm guessing the vertex shader is normalising the z value, but my debugging of the pixel shader appears to show fDepth values greater than 1.0.

I hope this helps.

[Hrenli]

The ESM issue happens on Intel and NVIDIA using OpenGL and DirectX.
I managed to reproduce in the ShadowMapDebugging sample and have attached the modified ShadowMapDebuggingGameState.cpp.

When I saw the picture I remembered I had similar issues with ESM in one of my scenes. Did not have the time to dig deeper, but now when you wrote it might be about non shadow-casting objects I realize I also had manually created mesh there with I think I had only one vao filled in and second vao just not touched (with shadowcasting set to false).

[dark_sylinc]

I have found why this happens.

Small note aside: Disabling shadow casting on the floor plane will cause a noticeable quality degration. This is a limitation of ESM.
This post is about a giant block of shadow in the far left corner of the plane.

The giant block of shadow is caused by outVs.posL0.z becoming higher than 1.0 (which is the clear value at which the depth is initialized).
This happens only with ESM because Z is calculated differently:

Code: Select all

//It's the same as (float4( worldPos.xyz, 1 ) * texViewMatrix).z
outVs.posL0.z = -(dot( worldPos.xyz, passBuf.shadowRcv[0].texViewZRow.xyz ) + passBuf.shadowRcv[0].texViewZRow.w);

This computation is not in [0; 1] range so it is brought to that range by doing:

Code: Select all

outVs.posL0.z = (outVs.posL0.z - passBuf.shadowRcv[0].shadowDepthRange.x) * passBuf.shadowRcv[0].shadowDepthRange.y;

Unfortunately, shadowDepthRange.y is not small enough and that's what's causing the bug. This value is computed from shadowCameraSetup->getMaxDistance() which is computed as the maximum distance from the shadow map camera to the furthest caster.
We'll have to calculate max distance in a different way for ESM.

The problem is that by disabling shadow casting on the floor, the receiver is much closer to the camera than the floor, so the floor becomes >1.
This doesn't happen with PCF because the math is different which guarantees the depth is in range [0;1]

I will have to think how to solve this problem, but in the meantime the workarounds I can suggest are:

• Clear the colour to something larger than 1. Not possible if format is PF_L16 though. It's possible if the format is is float16 or float32, but this can impact performance and or precision (float16 has distributes precision differently than PF_16, often for the worse in the case of ESM)
• Easy workaround: Just place a few caster objects periodically (even if empty!) to ensure the scene is always "enlarged"
• Hack Ogre to add some arbitrary padding on ShadowCameraSetup::mMaxDistance
• Just use PCF until the patch is ready

Thanks for the report, your research has been useful.

[Kaylx]

Thanks for confirming the fDepth issue.

I would change to PCF but they seem to be completely broken using Intel HD and most of my 'users' have Intel HD so it's not really an option.

I'll try the easy workaround. I was thinking about doing that sort of thing anyway as another issue I have is when objects are toggled on/off in the scene the quality of the shadows change.

How would recommend creating these empty/invisible objects?

[Kaylx]

I've also just noticed another issue.

HlmsBlendblock.mBlendChannelMask is not included in the comparison operator which means calling HlmsManager::getBlendblock doesn't work as intended.

Making the following change fixes things.

FYI i was trying to do the following but my object was still being drawn.

Code: Select all

    Ogre::HlmsBlendblock blendblock;
    blendblock.mBlendChannelMask = 0;
    Ogre::HlmsUnlitDatablock* datablock;
    datablock = static_cast<Ogre::HlmsUnlitDatablock*>( 
      hlmsUnlit->createDatablock( finalName, finalName,
                                  macroblock, blendblock,
                                  Ogre::HlmsParamVec() ) );

[dark_sylinc]

Thanks for the fix!

About the invisible objects: I was actually thinking in a custom renderable as in CustomRenderable example where "MyCustomRenderable" creates an empty VertexArrayObject without vertex nor index buffers. That should work (I think).

I would change to PCF but they seem to be completely broken using Intel HD and most of my 'users' have Intel HD so it's not really an option.

Ugh, Intel GL drivers are always a PITA. Is there a reason Intel users can't use D3D11? Their driver support is much better there.

[Kaylx]

Is there a reason Intel users can't use D3D11? Their driver support is much better there.

The render system is hard coded because for the longest time there were some GL only features. That's no longer the case so could try out DX on Intel.

Regarding the empty/invisible objects I managed to do it using a plane mesh and just changed the material's blend and macro block settings.
On the normal block I disable depth and colour writing and on the shadow caster block I make sure depth and colour writing are enabled.
I tried using an unlit block with the above settings first but it really broke the shadows so I changed to pbs with those settings and it seems to work ok.
So now I can switch objects on/off in the scene without affecting shadow quality.
I did try creating my own empty object using an item with a manual mesh (no geometry) and set the local aabb to something useful however the issue you mentioned in that other post still existed.
It did fix the quality issue I mentioned when turning objects on/off but the shadows didn't look great on the ground plane (i.e. too sharp just like in the other post above).

BTW what shadow technique would you recommend for large 'city scape' scenes?

[dark_sylinc]

The ESM bug where "a large shadow" appeared has been fixed.