[2.2] Self Shadowing errors

[SolarPortal]

Yeah ditsy moment haha

right, i'm running in DX11 not opengl, havent tested that yet

For those that want to know how to adjust the normal offset bias from an editor or something similar, i am using this at the moment, also works for adjusting pssmlambda on the fly too:

Code: Select all

	SkyString ShadowNode = chooseShadowNode(currentQuality, currentFilter);
	Ogre::CompositorShadowNodeDef* node = scene.renderMgr->getCompositorManager()->getShadowNodeDefinitionNonConst(ShadowNode);// findShadowNode(ShadowNode)->getDefinition();
	size_t numShadowDefinitions = node->getNumShadowTextureDefinitions();
	for (size_t i = 0; i < numShadowDefinitions; i++) {
		Ogre::ShadowTextureDefinition* texture = node->getShadowTextureDefinitionNonConst(i);
		texture->normalOffsetBias = value;
	} scene.renderMgr->getWorkspace()->generateWorkspace();

Controlling the constant bias. (currently having problems using this properly.)

Edit: Yes, looks fine in GL, could it be a -1 to 1 vs 0 to 1 or a row column(dx vs gl) thing of a matrix.

Edit2: Replaced the cxnstant bias setter

Edit3: Removed the code for the constant bias as you have to do it on the shadow camera, however there is no way that i can see of accessing the shadow camera data from the compositionShadownode

[Lax]

Hi all,

@dark_sylinc: Thanks for the terra fix! It now does not crash anymore.

But I now encounter strange shadows behavior. @SolarPortal, I don't know if its the same you encouter, so I made a video:

http://www.lukas-kalinowski.com/Homepag ... hadows.mp4

On my cute model called Jennifer, only the head does cast shadows. This was not the case using the master branch. The model is componsed just of one mesh. The models do cast shadows correctly. But when the camera does get some distance to the ground (watch especially the wall in the video) the shadows do start at an offset. There is something wrong.

Edit: I now used standard pbs workspace with models on a flat plane. See video:

http://www.lukas-kalinowski.com/Homepag ... adows2.mp4

In the next video something really strange happens:

http://www.lukas-kalinowski.com/Homepag ... adows3.mp4

Shadows are not casted on some object (are the objects to small? I'm using the unit 1 = 1 meter like newton phyics. Ogre uses 1unit = 100 meter. When I translate the small object away from ground the shadow will appear.

Jennifer still does only cast her head as shadow and when I translate the camera away from those platforms the shadows will vanish completely. But only on those plattforms.

@SolarPortal: Thanks for the code snippet. I will try it out soon.

Best Regards
Lax

[SolarPortal]

i get bad shadows on the second split and i sometimes get shadows on the front faces that face the light.
I haven't tried shadows on an object with such small scale yet.

[Lax]

Hi SolarPortal,

i get bad shadows on the second split and i sometimes get shadows on the front faces that face the light.
I haven't tried shadows on an object with such small scale yet.

Yes, I also get bad shadows quickely (second split). I think this new shadows calculation needs some more analysis.

Best Regards
Lax

[SolarPortal]

check the shadow depth bias that is stored on the material file has the new setting for the shadows constant bias as i found this is why my shadows went strange

Edit: Got good results with setting the depth bias on the material between 0.00050 > 0.0010
Got a good result with normal offset at value: 0.0000040
Doesnt look too disimilar to what i was playing with my version.
Shadows also scale well with the object size, my advice check the shadowmap bias on the materials itself.
Good job so far dark_sylinc

[dark_sylinc]

It is as SolarPortal says!

The scale of HlmsDatablock::mShadowConstantBias changed drastically!
It needs to be very low (the default right now is 0.001; it used to be 0.01).

The artifacts you're seeing in the video come from a very large constant bias.

Based on your feedback I pushed a change in the default values in order to achieve much higher backwards compatibility with existing materials. I don't want lots of users complaining about such silly thing.
Thus just upgrade to the latest commit and it should fix itself.

[SolarPortal]

confirmed, that last change works.

Here is the code to control the constant bias on a scene level:

Code: Select all

void ShadowManager::setGlobalBias(SkyFloat value, bool refreshWorkspace) {
	ShadowGlobalBias = value; 

	SkyString ShadowNode = chooseShadowNode(currentQuality, currentFilter);
	Ogre::CompositorShadowNodeDef* node = scene.renderMgr->getCompositorManager()->getShadowNodeDefinitionNonConst(ShadowNode);// findShadowNode(ShadowNode)->getDefinition();
	size_t numShadowDefinitions = node->getNumShadowTextureDefinitions();
	for (size_t i = 0; i < numShadowDefinitions; i++) {
		Ogre::ShadowTextureDefinition* texture = node->getShadowTextureDefinitionNonConst(i);
		texture->constantBiasScale = value;
	} scene.renderMgr->getWorkspace()->generateWorkspace();
}

So i would say that from my perspective and the way im running ogre that it is working.

There is that slight difference between openGL and DX11 on the ShadowmapDebugging sample which uses same settings between both renderers, so it should look the same. but im not seeing anything completely noticeable.

ESM seems to work using the 2.0 samples esm shadownode, but esm does not work on my shadow nodes for whatever reason, will open another topic about that

Good Job and thanks for the continued support!

Edit: Scratch that last image, wrong shadow map bias in material.. woops!

Edit2: Tested on trees with leaves and its working great and no artefacts on leaves facing the light and proper shadowing on under leaves

Edit3: Topic solved!

[zxz]

I'm also very interested in improving the shadow quality and have been trying out the changes from this thread. So it is very nice that this is being worked on!

Unfortunately I can't seem to get any usable results out of it yet. Do you set different biases for different PSSM splits? From the code above, it seems like SolarPortal doesn't. Using the same bias for all results in severe artifacts on the further splits for me. There was a mention in a commit message that the biasing is automatically amplified for further splits.

The default values result in peter-panning and shadows cut off over a meter from the object. So lowering the constant bias scale to 0.005-0.01 improves that, but introduces severe acne on further splits. Increasing the normal offset bias helps against that, but when it is increased enough to get rid of the acne, shadows are offset by a huge amount. I have to bring it to about 0.04 (several orders of magnitude), to remove the acne.

Where am I going wrong?

I am using GL3+ rendersystem.

[SolarPortal]

i cant for sure say that it works for opengl as i run using dx11.
however, most of the artefacts i have had can be sorted by setting the correct bias per material, ensuring the constant bias is around 0.1 and that the normal offset bias is also set to the correct value, i actually use the normal offset bias at 10x less than the default and have had scenes with no acne or peter panning.

Normal offset bias too high is going to add its own shadow problems.

Our units are 1 unit = 1 meter.

Are you also running with the front face culling? i cant remember if it was removed or just unused.

I do get the odd scene with a large peter panning offset, but offsetting biases tends to fix it.

By default in ogre, each split is running the 0.00004, which is why i set all splits to the same.

Edit: Its definitely the second split > where i get that offset effect that looks like peter panning but never on the first split.
but most of the time things are covering up bits at that range.
Even with 0 of constant bias and 0 on normal offset, the second split has an offset if running a larger tap on the PCF

First, comparison on the 2nd split with PCF2x2 and PCF6x6


Constant Bias set to default(left) and Constant bias default & Normal offset 10x less than the default(right)

Split 1 - with skyline defaults.( 0.05 & 0.0000040 )


Edit 2: Resolution of the shadow maps also changes things. Lower rez means you might get more acne and require a larger bias.
Higher resolution can reduce acne to the point of no artefacts and thus you can use a lower constant bias. ( 4096 splits make a difference. )
I found that the normal offset doesnt really have to change much.

Tip: large planes that give acne, simply turn off cast shadows.
Hope this gives some insight.

[dark_sylinc]

Unfortunately I can't seem to get any usable results out of it yet. Do you set different biases for different PSSM splits? From the code above, it seems like SolarPortal doesn't. Using the same bias for all results in severe artifacts on the further splits for me. There was a mention in a commit message that the biasing is automatically amplified for further splits.

Edit: Its definitely the second split > where i get that offset effect that looks like peter panning but never on the first split.
but most of the time things are covering up bits at that range.
Even with 0 of constant bias and 0 on normal offset, the second split has an offset if running a larger tap on the PCF

I added per-split offset so that these issues can be solved by hand. But based on these feedback it looks like Ogre needs to automatically set saner defaults for higher splits to reduce complaints.
I wish I knew that is exactly in the scene that we can use to auto adjust these values.

Btw zxz: are you able to post a few pics?

Edit 2: Resolution of the shadow maps also changes things. Lower rez means you might get more acne and require a larger bias.
Higher resolution can reduce acne to the point of no artefacts and thus you can use a lower constant bias. ( 4096 splits make a difference. )
I found that the normal offset doesnt really have to change much.

That is definitely a bug, we must be incorrectly accounting shadow map resolution.

[SolarPortal]

I added per-split offset so that these issues can be solved by hand. But based on these feedback it looks like Ogre needs to automatically set saner defaults for higher splits to reduce complaints.

Agreed.

In the meantime here is a quick bit of code for you all that should be easy to adapt.
It allows the control of each split and the global as a whole.
Setting the global bias sets all splits too, so set the splits after the global.
Same for normal offset.

Code: Select all

void ShadowManager::updateGlobalBias(bool refreshWorkspace)
{
	SkyString ShadowNode = chooseShadowNode(currentQuality, currentFilter);
	if (ShadowNode == "") return;
	Ogre::CompositorShadowNodeDef* node = scene.renderMgr->getCompositorManager()->getShadowNodeDefinitionNonConst(ShadowNode);// findShadowNode(ShadowNode)->getDefinition();
	size_t numShadowDefinitions = node->getNumShadowTextureDefinitions();
	for (size_t i = 0; i < numShadowDefinitions; i++) {
		Ogre::ShadowTextureDefinition* texture = node->getShadowTextureDefinitionNonConst(i);

		if (i == 0) {
			texture->constantBiasScale = ShadowGlobalBias_PSSMSplit[0];
		}
		else if (i == 1) {
			texture->constantBiasScale = ShadowGlobalBias_PSSMSplit[1];
		}
		else if (i == 2) {
			texture->constantBiasScale = ShadowGlobalBias_PSSMSplit[2];
		}
		else {
			texture->constantBiasScale = ShadowGlobalBias;
		}

	} scene.renderMgr->getWorkspace()->generateWorkspace();
}

void ShadowManager::setGlobalBias(SkyFloat value, bool setAll, bool updateWorkspace) {
	ShadowGlobalBias = value; 
	if (setAll) {
		ShadowGlobalBias_PSSMSplit[0] = value;
		ShadowGlobalBias_PSSMSplit[1] = value;
		ShadowGlobalBias_PSSMSplit[2] = value;
	}
	if (updateWorkspace)updateGlobalBias(true);
}


void ShadowManager::setGlobalBias_PSSMSplit(size_t idx, SkyFloat value, bool updateWorkspace) {
	ShadowGlobalBias_PSSMSplit[idx] = value;
	if (updateWorkspace)updateGlobalBias(true);
}
SkyFloat ShadowManager::getGlobalBias_PSSMSplit(size_t idx) {
	return ShadowGlobalBias_PSSMSplit[idx];
}

void ShadowManager::updateNormalOffset(bool refreshWorkspace)
{
	SkyString ShadowNode = chooseShadowNode(currentQuality, currentFilter);
	if (ShadowNode == "") return;
	Ogre::CompositorShadowNodeDef* node = scene.renderMgr->getCompositorManager()->getShadowNodeDefinitionNonConst(ShadowNode);// findShadowNode(ShadowNode)->getDefinition();
	size_t numShadowDefinitions = node->getNumShadowTextureDefinitions();
	for (size_t i = 0; i < numShadowDefinitions; i++) {
		Ogre::ShadowTextureDefinition* texture = node->getShadowTextureDefinitionNonConst(i);
		if (i == 0) {
			texture->normalOffsetBias = ShadowNormalOffset_PSSMSplit[0];
		}
		else if (i == 1) {
			texture->normalOffsetBias = ShadowNormalOffset_PSSMSplit[1];
		}
		else if (i == 2) {
			texture->normalOffsetBias = ShadowNormalOffset_PSSMSplit[2];
		}
		else {
			texture->normalOffsetBias = ShadowNormalOffset;
		}
	}
	if (refreshWorkspace) { scene.renderMgr->getWorkspace()->generateWorkspace(); }
}

void ShadowManager::setNormalOffset(SkyFloat value, bool setAll, bool updateWorkspace) {
	ShadowNormalOffset = value; 
	if (setAll) {
		ShadowNormalOffset_PSSMSplit[0] = value;
		ShadowNormalOffset_PSSMSplit[1] = value;
		ShadowNormalOffset_PSSMSplit[2] = value;
	}
	if (updateWorkspace)updateNormalOffset(true);
}

void ShadowManager::setNormalOffset_PSSMSplit(size_t idx, SkyFloat value, bool updateWorkspace) {
	ShadowNormalOffset_PSSMSplit[idx] = value;
	if(updateWorkspace)updateNormalOffset(true);
}
SkyFloat ShadowManager::getNormalOffset_PSSMSplit(size_t idx) {
	return ShadowNormalOffset_PSSMSplit[idx];
}

SkyFloat ShadowManager::getGlobalBias() { return ShadowGlobalBias; }
SkyFloat ShadowManager::getNormalOffset() { return ShadowNormalOffset; }

Btw zxz: are you able to post a few pics?

I have one scene at the moment that i am trialling with to test whether the split seperation control helps in any way as i have a sort of mesh terrain with a track and it gets the grey shadows from second split all over like what zxz was mentioning and it takes high levels of biasing at which point the shadows become equally unusable. If i can get some screens for you i will, but im hoping the above code will let me solve it.

That is definitely a bug, we must be incorrectly accounting shadow map resolution.

perhaps... where abouts could it be? As the MJP demo did divide the normal offset function by the cascade split. Not sure if that could attribute anything.

Edit: Updated code above to fix a couple bugs

[zxz]

I might be able to provide some screenshots later.

For now I am trying to debug some inconsistencies between the amount of peter-panning I'm getting with the same biasing parameters in different scenes. In one largish scene I get huge peter-panning with the default parameters, which I can't easily recreate from scratch in a minimal test scene (from which I planned to provide screenshots). I want to figure out what's going on with that before I make more claims.

It seems like removing or hiding some larger meshes won't give immediate effect on the shadows, but after a complete reload it does. That makes me think some bounds don't get recomputed as they should. That's making it a bit harder to get reproducible results and figure out exactly what my problems are.

One thing I noticed while debugging is that the shaders' shadowRcv[n].shadowDepthRange always is [0, 1e-6]. That doesn't seem right. It doesn't seem to be used for the biasing or directional lights in general though, so my problem probably lies elsewhere.

[edit] In general, I need some kind of increased constant biasing for each split, something like 0.01, 0.02, 0.05, 0.3 to get decent results. These numbers were just arrived at experimentally, trying to keep them as low as possible without introducing acne.

[zxz]

It seems like stable splits are one of the causes of my shadow problems. I get massively more peter-panning in stable splits compared to unstable splits, with the same biasing settings. I have only looked at this briefly, but the default parameters seem to work pretty well now, with no stable splits.

Edit: I guess I should have read the thread more closely

[Lax]

Hi all,

I also can confirm, that shadows now are correct

Best Regards
Lax

[SolarPortal]

on my scene, i can get rid of the floor artefact by setting splits with this const bias: (split0)0.01, (split1)0.02/0.03, (split2)0.08
I do think it will be down to the scene, the resolution(which is a bug), and the lighting angles. Low angle lights produce the most errors.

Edit: At higher light angles, i found changing the normal offset per split better than the const bias per split.

[dark_sylinc]

For now I am trying to debug some inconsistencies between the amount of peter-panning I'm getting with the same biasing parameters in different scenes. In one largish scene I get huge peter-panning with the default parameters, which I can't easily recreate from scratch in a minimal test scene (from which I planned to provide screenshots). I want to figure out what's going on with that before I make more claims.

Mmmm it looks like my decision was a bad one.
The current caster code looks like this:

Code: Select all

float shadowConstantBias = -uintBitsToFloat( worldMaterialIdx[inVs_drawId].y );

However it originally looked like this:

Code: Select all

float shadowConstantBias = -uintBitsToFloat( worldMaterialIdx[inVs_drawId].y ) * passBuf.depthRange.y * passBuf.depthRange.y;

Without the squared multiplication the larger the range, the larger the bias becomes. This nicely adapts to larger splits which need increasing biasing.
However I suspected it would cause unpredictability in scenes because the biasing would now depend on scenes.

I thought it wouldn't matter, but if you noticed and warranted research from you that means my solution was bad, and we should re-incorporate the squared multiplication. The increase of constant biasing should probably be done more explicitly, rather than be cryptically hidden behind math.

[zxz]

Well... most of my issues turned out to be self-inflicted (as is too often the case) and should not have any bearing on the discussion. I will describe the issue briefly for completeness, but it probably won't be of interest to anybody else. Long story short, we update our workspaces manually for increased control, and manually make some calls that Ogre::Root and Ogre::CompositorManager otherwise do. When porting from 2.1 to 2.2, we accidentally lost the bumping of the frame number, which meant that some shadow node calculations were being reused where they should not. That led to a lot of unnecessary confusion on my part. Sorry about that. Due to that bug, disabling some large shadow casters didn't affect the results as they should have. Together with using only stable splits, it was very difficult to get good results. Keeping the last 1-2 splits unstable really helps.

Now my results are probably much more in line with what others report. I can get good results with some minor bias tweaking.

The amount of peter-panning does depend on the size of shadow casters, and large casters will lead to larger panning with the same bias. Using stable splits amplifies this effect by a large amount. That will be one more variable to take into consideration when configuring the biasing. It's not quite as simple as one biasing fits all, but that might be too much to ask for

[dark_sylinc]

OK!

I pushed a new update (which I only tested on Linux so far).

The commit explains everything that has been done.

Notably for the testers in this thread:

• Default value of constantBiasScale changed from 0.1 to 1.0
• The 'scale' of constantBiasScale (increase your current values by 10x) and normalOffsetBias (increase your current values by 4194304x) have drastically changed
• Fix normal offset bias heavily dependent on shadow map resolution (as reported by SolarPortal)
• Added autoConstantBiasScale and autoNormalOffsetBiasScale, which as zxz described they account the size of the casters. Turns out accounting 'depth range' was sort of working because it's indirectly related and proportional to the size of the shadow casters. Now we consider this size instead of the depth range, resulting in more consistent and stable results

Overall you might still have to tweak some values beyond their defaults, but I hope the need becomes much lower than with the previous result.

Also I fixed a bug with stable PSSM splits, and added shadow pancaking (used with unstable PSSM).
Shadow pancaking increases the chance of being able to replace a 32-bit depth buffer for a 16-bit one, thus halving memory consumption (and bandwidth) on shadow mapping.

I hope everything goes well so we can merge this to master.

[Lax]

Hi dark_sylinc,

I tried to pull the newest changes. But I always get merge conflicts in hlms and source files. I even reset local changes (which I did not have, because I'm using Ogre only for readonly). I even deleted all local source files. But even then I get merge conflicts. Do you have any idea, how to solve this issue?

Best Regards
Lax

[zxz]

I am guessing that your problem comes from dark_sylinc rebasing the shadow_bias_improve branch on top of master. You can reset your branch to the origin's location with:

Code: Select all

$ git reset --hard origin/shadow_bias_improve

The above fix assumes that you have the branch shadow_bias_improve checked out. Make sure you don't have local changes first, as they would be lost.

[SolarPortal]

Thanks will check it out after my current work is done

Edit: @dark_sylinc: Just wondering if this page will help at all: https://catlikecoding.com/unity/tutoria ... l-shadows/ as it shows quite a few techniques we have already and a couple more like alpha blended shadows etc...

[Lax]

git reset --hard origin/shadow_bias_improve

Thanks zxz, that helped.

[SolarPortal]

Currently looking into it, but currently i only seem to be able to have 1 point light with shadows now? Is anyone else getting the same.
My shadow node is set up for 20 shadows lol

The shadow changes do seem to work though for the directional light and the offsets on spot and point lights are equally good

Edit: It appears that the point lights are switching to shadows mode when lots are in the level, but the shadow does not appear.
Spot lights on the other hand work fine

Heres an example which you can see which lights are active for the shadows at that moment due to the attenutation differences:


(had my tablet handy, so used that )

Key:
S = Shadowing (Correct)
NS = No Shadow (Correct)
S/NS = Shadowing but no shadow (Wrong)

Hope this helps to explain the problem.

[dark_sylinc]

Hope this helps to explain the problem.

It did! Thanks! Fixed.

The problem being localized to point lights made me remember I had to change 'counter' to 'value' but then the code was modified again and that little change had to be reverted, but I forgot. When you mentioned it, that line was my first suspect.

Cheers and thanks for pointing it! (pun intended)
Matias

[SolarPortal]

haha

That works for me, would have been quite pointless(:P) having shadows maps active without the fix
I'm finding standard settings to be quite good so far!

Good job!