Shadow improvements

[dark_sylinc]

You've written:

Code: Select all

shadow_map 5 atlas uv 0.5 0.0 1.0 0.25 light 3

However that won't work because it makes your spotlight span in UV range 0.5 through 1.5; which is not valid (hence half a circle!)
The syntax is:

Code: Select all

shadow_map 5 atlas uv left top width height light 3

Change it to:

Code: Select all

shadow_map 5 atlas uv 0.5 0.0 0.5 0.25 light 3

And it should work (as long as you don't have more overlaps or out of bounds UV specification)

Btw: You no longer need to set the PSSM's directional light brightness to 0. If you disable the light, we will now handle it correctly without crashing (although it would trigger a shader recompilation so for an editor perhaps it's still benefitial to set the brightness to 0 to avoid the stall)

[SolarPortal]

Thanks for the quick response. That's makes total sense, I was thinking in uv positions rather than width and height for the second uv set.

I'll try tomorrow.Thanks again

Edit: Made the change and the third shadow is working correctly. Thanks for the information, now to get the rest of the shadows in

[SolarPortal]

ok, i think i have found a bug in the OgreHlms.cpp - preparePassHashBase() function which is what caused my PSSM 2nd split plane(shadow_map 1) to muck up and disappear when shadow_map 10 is added.

When the 10th shadow map is used, the "basePropSize" resets the name to only having index "hlms_shadowmap1" as the "propName" instead of "hlms_shadowmap10".

So for example at line: 2141, the propName variable is set to hlms_shadowmap10 correctly, but on the next lines when the propName is resized to have the other appends,
it resets to "basePropSize" which has a size of 15, removing the 0 of the 10 in the "propName" leaving "hlms_shadowmap1" instead.

This resets the propName to hlms_shadowmap1 which then overwrites the second split plane(shadow_map 1) of the pssm instead of handling the 10th shadow map correctly.

For example, this happens at 2148:

Code: Select all

if( shadowTexDef->uvOffset != Vector2::ZERO ||
    shadowTexDef->uvLength != Vector2::UNIT_SCALE )
{
    propName.resize( basePropSize ); // The "hlms_shadowmap10" is changed into a "hlms_shadowmap1" on propName leaving broken pssm shadows.
    propName.a( "_uvs_fulltex" );
    setProperty( propName.c_str(), 1 );
}

It then happens to all calls after whenever the "propName" is resized back to the "basePropSize".
e.g. lines: 2156, 2159, 2164, 2167, 2173, 2176, 2181, 2184, 2188, 2195, 2200, 2203, 2208, 2211

I propose that we add say a temp var at the top around line 2139 which will have a basePropsize of 16 instead of 15 if the shadow map index enters a double figure to stop the pssm from breaking.
Otherwise, we can only use shadow_maps 0-9.

I hope this makes sense and that this helps you to be able to fix it
If you dont have time, i will try to come up with a solution tomorrow and post back

[dark_sylinc]

Fixed. Thanks!

Btw there is a known bug where UV offsets that have leading zeroes won't work, e.g. 0.102 works, but 0.056 won't work because we'll output 0.56 instead (a 0 was lost). Shouldn't be much of an issue and low priority for me, but it's worth pointing it out.
You shouldn't encounter this issue unless your resolution is huge and you pack A LOT of shadow maps together. Workaround would be to use more than one atlas. i.e. You would need to pack more than 10 shadows maps of the same size along the same axis (10x10 = 100 shadow maps per atlas) to encounter this issue.

[SolarPortal]

Awesome. Thanks for the quick fix.
Will test tomorrow
Thanks for extra information.
Edit: Works great now. Thanks again.

[SolarPortal]

Do you get the same results as me, if you have no lights in your scene then it crashes in GL with this error:

OGRE EXCEPTION(5:ItemIdentityException): Parameter called texShadowMap0 does not exist. in GpuProgramParameters::_findNamedConstantDefinition at ...OgreMain\src\OgreGpuProgramParams.cpp (line 2214)

or it happens after deleting all lights once they had been in the scene casting shadows.
apart from this error, the shadows seem very stable so far.

[dark_sylinc]

I was able to repro the bug. That's definitely a bug.
Edit: Done.

[SolarPortal]

Yes, definitely fixed. Great work!

Also tested on some of the samples like the instant radiosity and these are also more stable on GL now.

[SolarPortal]

New bug found in the "ShadowMapping_piece_ps.any" file at line 338.
If i use a mesh such as vegetation that has alpha rejection on the leaves, then it was crashing since it was adding this to the end of the PS:

Code: Select all

outDepth = inPs.depth;

when the

Code: Select all

float depth

was not added in the @piece( VStoPS_block ).

To get around this, i compared to the older PBS shaders and noticed that the line on "ShadowMapping_piece_ps.any" at 338 was potentially broken.
Changing it from:

Code: Select all

	@property( !hlms_shadowcaster_point )

to

Code: Select all

	@property( (!hlms_shadow_uses_depth_texture || exponential_shadow_maps) && !hlms_shadowcaster_point )

allowed the vegetation to render correctly with a directional, point & spot casting shadows simultaneously.
Just wanting to check that this is the correct fix

Edit: Forgot to add the error from the log: here it is:

19:20:21: GLSL compile log: 537165824PixelShader_ps
0(167) : error C1009: "depth" is not member of struct "block"
19:20:21: OGRE EXCEPTION(3:RenderingAPIException): Fragment Program 537165824PixelShader_ps failed to compile. See compile log above for details. in GLSLShader::compile at ...\RenderSystems\GL3Plus\src\GLSL\OgreGLSLShader.cpp (line 308)

[Feanor16]

When i use your new shadow node model, it make freeze my computer...

Code: Select all

compositor_node_shadow ShadowMapDebuggingShadowNode
{
	technique pssm

	texture pssm0 1024 1024 PF_D32_FLOAT
	texture pssm1 512 512 PF_D32_FLOAT
	texture pssm2 512 512 PF_D32_FLOAT

	texture spot0 1024 1024 PF_D32_FLOAT
	texture spot1 1024 1024 PF_D32_FLOAT

	num_splits		3
	pssm_lambda		0.95
	shadow_map 0 pssm0 light 0 split 0
	shadow_map 1 pssm1 light 0 split 1
	shadow_map 2 pssm2 light 0 split 2

	technique focused
	shadow_map 3 spot0 light 1
	shadow_map 4 spot1 light 2

	shadow_map_target_type directional
	{
		shadow_map 0 1 2
		{
			pass clear
			{
				colour_value 1 1 1 1
			}
			pass render_scene
			{
			}
		}
	}

	shadow_map_target_type directional spot
	{
		shadow_map 3 4
		{
			pass clear
			{
				colour_value 1 1 1 1
			}
			pass render_scene
			{
			}
		}
	}
}

[dark_sylinc]

When i use your new shadow node model, it make freeze my computer...

Does it happen with the samples?
What's your...?:

• CPU
• OS
• GPU
• GPU Driver version

If it's Windows, is it freeze or TDR (Timeout Detection Recovery) or BSOD?

[Feanor16]

i'm working on Ubuntu 16.04
my CPU is a Intel(R) Core(TM) i7-3960X CPU @ 3.30GHz
using a NVIDIA GeForce GTX 680 version a1

Well, i just replaced my shadow Node by the shadow node sample contained in : [...]/OGRE/Media/2.0/scripts/Compositors/ShadowMapDebugging.compositor

to my workspace.compositor

[xrgo]

I am definitely going to use that... I would like to use it this way:
one big static shadow covering the whole scene... maybe 8k? maybe resolution its not that important anymore since it will be static, so it will not flick.
and one pssm shadow for dynamic objects.
the thing is... how can achieve a blend/merge for this 2 shadowmaps? is it possible to generate 2 shadowmaps with different techniques form 1 unique light (the sun!) and then both maps should be multiplied right? its done automatically?

I just want to say I (kind of...) achieved this, in a no so clean way, but it works =D! what I did is:
have a directional light, and use a normal pssm shadows, but only for the dynamic objects, using visibility mask.
have a spot light veeeeeery far away, pointing at the same direction as the previous light and make sure its covering the whole scene with its cone, and use a static focused shadow, but only for static objects, again using visibility mask.
then modify the Pbs shaders to multiply the directional light shadow with the spot light shadow and apply it to the directional light's "darkenwithshadow" and omit the spot light contribution on the objetcs.

and it works! shadows looks great, the static one don't flicker so it looks good and static and dynamic objects interact perfectly. In my very old laptop I got better framerate... but I still need to test on a better machine in the office (nvidia 1070)... hopefully with something like this I will finally get 90fps in VR

Now the thing is:
1) It would be nice if we can use the directional light's position to fix the position of the shadow to the scene when using focused (right now, if you use focused and static the shadow moves with the camera)... kind of what blender does (in Blender Game renderer, select the sun lamp and check "Show Shadow Box") that way I don't have to use a very far away spotlight to make it look like a directional light.
2) It would also be nice to integrate in the pbs implementation the use of a light for just to generate a shadow so it can be mixed/merged/mulltipied with the one generated by another light.... OR (if (1) gets done...) a way so a light can generate 2 shadows maps defined by compositor shadow node. So for example in the shadow node... for a directional light I can have a pssm shadowmaps and a focused one, so I can set the focused one as static and be used (again.. if (1) gets done) for static objects =D!

Thank you so much for this feature!

[xrgo]

YES! I tried in very troublesome scene, with an nvidia 1070, and I went from 130fps to 170fps with this technique! yuhu!
pleeeaassseee can we have those features I asked, so we can use this technique more easily without breaking stuffs :3 <3... I am pretty sure I am not the only one who could benefit from this!
thank you!!

[TaaTT4]

I want it too!

xrgo, can you maybe upload a video that shows this technique in action?

Are you using Matias terrain?
Do you think it's possible to integrate your solution with its shadows?

[xrgo]

I've made a Gif:



Yellow objects are static, white ones are dynamic, I exaggerated the resolution difference in the shadows, lower res is the static one, generated by a very far spotlight, higher res is a pssm directional light shadow... as you can see, the moving static object wont update its shadow, but that's ok because it should not really move, I just put it to show that the shadow is really static

[TaaTT4]

I've made a Gif:



Yellow objects are static, white ones are dynamic, I exaggerated the resolution difference in the shadows, lower res is the static one, generated by a very far spotlight, higher res is a pssm directional light shadow... as you can see, the moving static object wont update its shadow, but that's ok because it should not really move, I just put it to show that the shadow is really static

It's f***ing cool!

[renyingzhi1006043]

Hello dark_sylinc：
how add vsm in Ogre2.1？
esm is not so good .

[dark_sylinc]

Hello dark_sylinc：
how add vsm in Ogre2.1？
esm is not so good .

Hi, VSM (Variance Shadow Mapping) is not implemented, and it was considered but ESM was favoured because VSM has many issues.

ESM isn't perfect but it usually delivers good quality; and often if it produces poor results it can be solved or workarounded. What is the problem you're having? (could you post some screenshots?)

[renyingzhi1006043]

Well,I'm using 2.1 make an indoor DEMO.
I've add PBS、 HDR、PCC and InstantRadiosity In DEMO,
PBS: PBS Very powerful , however I found Cannot change diffuse uv.
IR：IR looks great,Very great.
SSR: not very well, I Use PCC instead reflected, SSR of Panda3D is very good.
SSAO: not support FSAA，have not been added in DEMO.
Soft Shadow:ESM Bad performance,So I want to try using VSM.
PlanarReflections:Next, add in.

When I've finished this DEMO, I post screenshots.

thanks dark_sylinc.
Cheers!

[TaaTT4]

I'm moving my engine to the new shadow system (atlas shadow maps, static lights and so on).

I have three doubts that I'd like to clarify before making bad design choices.

• Is there a way to clear just the shadow map region of an atlas texture?
  In the OGRE sample, all atlas is cleared and then all its shadow maps are re-generated.
• If I want to avoid a dynamic spotlight ping-pong (due to another nearest light), the correct way to go is to treat it like a static spotlight (setLightFixedToShadowMap) and marking it dirty on every frame (setStaticShadowMapDirty).
  Am I right?
• What's the purpouse of shadow_map_full_viewport?
  When and where is it needed?

[dark_sylinc]

• Is there a way to clear just the shadow map region of an atlas texture?
  In the OGRE sample, all atlas is cleared and then all its shadow maps are re-generated.

Short version:
No, unless you run a pixel shader that writes to depth and clears it by hand (not recommended, prevents fast Z clears and other stuff like Hi-Z)
That's why the static shadow map sample uses two atlases, one for dynamic lights, another for static lights.

Long version:

• D3D11 does not allow clearing subregions.
• GL does, but it is unknown how it works with subregions (i.e. whether it runs a pixel shader and clears by hand). Most likely GL runs fast Z clears on block boundaries and then manually sets blocks that are half covered. e.g. if internally the GPU works with 4x4 blocks of Hi-Z, and you want to clear a subregion of 1026x4; the first 256 blocks of 4x4 do ok, but the last block (the 257th) needs to be done by hand because only half of the pixels in the block are used. Also, don't assume all GPUs work in blocks of 4x4, it was just an example.
• Metal encourages clearing subregions on Mobile, because of how TBDR GPUs work.
• D3D 11.1 added a way to clear subregions, but it needs the 11.1 update (I do not know if it works in Windows 7 btw).
• Ogre 2.2 allows using clearing of subregions when available and/or when it's faster; but if D3D11.1 isn't available obviously in D3D11 it can't clear subregions. Note that 2.2 is WIP.

• If I want to avoid a dynamic spotlight ping-pong (due to another nearest light), the correct way to go is to treat it like a static spotlight (setLightFixedToShadowMap) and marking it dirty on every frame (setStaticShadowMapDirty).
  Am I right?

Yes.

• What's the purpouse of shadow_map_full_viewport?
  When and where is it needed?

[/list]

When you create a pass that works on a shadow map, the manually-specified viewport settings will be overriden to fit the region in the atlas (i.e. Ogre will ignore the "viewport" setting).
Setting this to true allows you to manually specify the viewport to whatever you'll like, which can be larger or smaller than the UV region it should be writting to.
For example the StaticShadowMaps sample uses this setting to check that if the first static shadow map got set as dirty (which is shadow map 3) then clear the whole atlas:

Code: Select all

//Clear the static atlas first. Do this inside a shadow_map_target_type block so that it only
//gets executed if shadow_map 3 is flagged as dirty (instead of clearing every frame).
shadow_map_target_type directional spot point
{
	shadow_map 3
	{
		pass clear
		{
			colour_value 1 1 1 1
			shadow_map_full_viewport yes
		}
	}
}

If you dirty shadow map 4 instead of 3; CompositorShadowNode::setStaticShadowMapDirty will automatically flag shadow_map 3 as dirty as well (unless includeLinked is set to false)
Basically, when one static light gets dirty, we have to treat all other static lights in the same atlas as dirty.

If we were to ommit the "shadow_map_full_viewport yes", then in D3D11 the whole atlas would be cleared but in GL only the subregion of shadow_map 3 would be cleared, leading to wrong results in the rest of the shadow maps sharing the atlas in GL.