Shadow bias problem Direct3D9 Topic is solved

Problems building or running the engine, queries about how to use features etc.
Post Reply
rpgplayerrobin
Gnoll
Posts: 617
Joined: Wed Mar 18, 2009 3:03 am
x 353

Shadow bias problem Direct3D9

Post by rpgplayerrobin »

Ogre Version: 1.12.13
Operating System: Windows 10
Render System: Direct3D9

Hey!

I am having some problems regarding shadow-casting spotlights in Direct3D9 that has existed since 1.11.2.

Shadows are sometimes visible on surfaces that should really not have shadows.
To fix this, shadow bias is used, but in Direct3D9, this bias act quite strange for spotlight shadows.

Here is a video showing the problem (first part shows the problem, the second part at 0:09 shows the fix with my very ugly solution):

I am using my own written shaders for shadow casting and shadow receiving, and both versions of shaders looks exactly the same on Direct3D9 and Direct3D11 (apart from Direct3D11 is edited to use "Texture2D" and "SamplerState" instead of just "sampler" of course).

The problem does not exist in Direct3D11, only in Direct3D9.
The problem does not exist for my shadow-casting directional light either, only for spotlights.

I have tried different attenuation (which means far plane distance) and setSpotlightNearClipDistance with the spotlight, but it never changes the results (apart from not rendering the shadows as far of course).

My code is basically exactly the same as the one here:
https://wiki.ogre3d.org/Depth+Shadow+Mapping
I have also tried with the code when using LINEAR_RANGE, but it does not help.

I am not using a custom shadow camera or anything like that.

My current very ugly solution to this problem in Direct3D9 is to make the shadow bias 70 times higher for spotlights in the shaders (also shown in the video).
There are still some artifacts but it is working pretty much 99% of the time, and the shadows are strangely enough not that far off (since shadow bias makes them "skip" the start of the shadows).
Doing this on the directional light makes shadows be cut off pretty horribly at the start since the bias is too large, so I am wondering why it is working differently there compared to spotlights.
On Direct3D11 the shadow bias for the directional light and the spotlight is exactly the same and it works correctly there without having to make it 70 times higher for spotlights.

Here are the minimal shaders that makes it reproducable:
ShadowCaster:

Code: Select all

uniform float4x4 worldViewProj;
uniform float4 texelOffsets;

struct VS_OUTPUT
{
	float2 oDepth : TEXCOORD0;
};

VS_OUTPUT main_vs(float4 position : POSITION,
				  out float4 oPosition : POSITION)
{
	VS_OUTPUT Out;
	oPosition = mul(worldViewProj, position);
	oPosition.xy += texelOffsets.zw * oPosition.w;

Out.oDepth.x = oPosition.z;
Out.oDepth.y = oPosition.w;
return Out;
}

float4 main_ps(float2 depth : TEXCOORD0) : COLOR0
{
	float finalDepth = depth.x / depth.y;
	float4 color = float4(finalDepth, 1, 1, 1);
	return color;
}

ShadowReceiver:

Code: Select all

float4x4 worldMatrix;
float4x4 texViewProj1;
float4x4 modelViewProj;

struct VS_OUTPUT
{
	float3 oVertexPos : TEXCOORD0;
	float4 oShadowUV1 : TEXCOORD1;
};

VS_OUTPUT main_vs( float4 position : POSITION,
				   out float4 oPosition : POSITION )
{
	VS_OUTPUT Out;

float3 worldPos = mul(worldMatrix, position).xyz;

oPosition = mul(modelViewProj, position);

Out.oVertexPos = worldPos;
Out.oShadowUV1 = mul(texViewProj1, float4(Out.oVertexPos, 1));

return Out;
}

float computeShadow(sampler targetShadowMap, float4 targetShadowUV)
{
	targetShadowUV.xyz = targetShadowUV.xyz / targetShadowUV.w;

float shadowBias = 0.000005;
shadowBias += shadowBias * 70.0; // We need this on Direct3D9 for spotlights for some reason

float shadowTotalOpacity = 0.0;
float shadowSample = tex2D(targetShadowMap, targetShadowUV.xy).x;
if(shadowSample + shadowBias < targetShadowUV.z)
	shadowTotalOpacity += 1.0;

return shadowTotalOpacity;
}




float computeSpotlight( float3 LightToVertexNorm, float3 lightdirection, float3 spotlightparams )
{
	float cosSpotLightAngle = saturate(dot(LightToVertexNorm, lightdirection));
	return pow(saturate(cosSpotLightAngle - spotlightparams.y) / (spotlightparams.x - spotlightparams.y), spotlightparams.z);
}



float4 lightPosition1;
float4 lightSpotParams1;
float4 lightDirection1;

sampler ShadowMap1 : register(s6); // Looks strange, but it is at that spot in the material

float4 main_ps(float3 position : TEXCOORD0,
			   float4 shadowUV1 : TEXCOORD1) : COLOR0
{
	float4 color = float4(1,1,1,1);

float shadowOpacity = 1.0 - computeShadow(ShadowMap1, shadowUV1);
color.xyz *= shadowOpacity;

color.xyz *= computeSpotlight(normalize((position - lightPosition1.xyz)), lightDirection1.xyz, lightSpotParams1.xyz);

return color;
}

And here is the initialization code for the shadows:

Code: Select all

// Set the shadow values
app->m_SceneManager->setShadowTechnique(SHADOWTYPE_TEXTURE_MODULATIVE_INTEGRATED);
if(tmpCasterMaterial) // This is the DepthShadowmapCaster shader material
	app->m_SceneManager->setShadowTextureCasterMaterial(tmpCasterMaterial);
app->m_SceneManager->setShadowTextureSelfShadow(true);

int tmpNumberOfShadowTextures = 2;

// Create the shadow textures
app->m_SceneManager->setShadowTextureCount(tmpNumberOfShadowTextures);
int tmpShadowTextureSize = 2048;
for (int i = 0; i < tmpNumberOfShadowTextures; i++)
{
	if (i > 0)
		tmpShadowTextureSize = 512;
	app->m_SceneManager->setShadowTextureConfig(i, tmpShadowTextureSize, tmpShadowTextureSize, PF_FLOAT32_R);
}

app->m_SceneManager->setShadowUseInfiniteFarPlane(false);
app->m_SceneManager->setShadowCasterRenderBackFaces(false);
app->m_SceneManager->setShadowFarDistance(12.0f);

How can the result be completely different when both Direct3D9 and Direct3D11 are using the same shaders and same initialization (well, apart from the necessary Direct3D11 texture fetch changes in the shaders)?

rpgplayerrobin
Gnoll
Posts: 617
Joined: Wed Mar 18, 2009 3:03 am
x 353

Re: Shadow bias problem Direct3D9

Post by rpgplayerrobin »

Oh, and by the way, there is a bug in Ogre (even in the latest version) that makes spotlight shadows kind of weird.
If you rotate a spotlight until it is upside down, the shadow texture flips. This creates some unwanted bugs.

To fix it:

OgreShadowCameraSetup.cpp:
From:

Code: Select all

Vector3 up = Vector3::UNIT_Y;
// Check it's not coincident with dir
if (Math::Abs(up.dotProduct(dir)) >= 1.0f)
{
	// Use camera up
	up = Vector3::UNIT_Z;
}
texCam->getParentNode()->setOrientation(Math::lookRotation(dir, up));

To:

Code: Select all

if (light->getType() == Light::LT_DIRECTIONAL)
{
	Vector3 up = Vector3::UNIT_Y;
	// Check it's not coincident with dir
	if (Math::Abs(up.dotProduct(dir)) >= 1.0f)
	{
		// Use camera up
		up = Vector3::UNIT_Z;
	}
	texCam->getParentNode()->setOrientation(Math::lookRotation(dir, up));
}

OgreShadowRenderer.cpp:
From:

Code: Select all

texCam->getParentSceneNode()->setDirection(light->getDerivedDirection(), Node::TS_WORLD);

To:

Code: Select all

Node* tmpParentSceneNode = light->getParentNode();
if (tmpParentSceneNode)
	texCam->getParentSceneNode()->setOrientation(tmpParentSceneNode->_getDerivedOrientation());
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Shadow bias problem Direct3D9

Post by paroj »

depth bias is expressed in depth-buffer units. What they mean depends on the depth buffer-format and how far stuff is away from the near plane.
Make sure the both depth format and the projection matrix end up similar in D3D9 and D3D11.

Maybe you are rendering depth through FFP in D3D9? For depth bias the caster material is important.

paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Shadow bias problem Direct3D9

Post by paroj »

If you rotate a spotlight until it is upside down, the shadow texture flips. This creates some unwanted bugs.

can you elaborate on that? The shadow matrix adapts with the shadow texture, so the scene orientation should not matter.

On the other hand having coincident up and dir gives you a wrong rotation matrix..

rpgplayerrobin
Gnoll
Posts: 617
Joined: Wed Mar 18, 2009 3:03 am
x 353

Re: Shadow bias problem Direct3D9

Post by rpgplayerrobin »

paroj wrote: Wed May 18, 2022 12:32 pm

depth bias is expressed in depth-buffer units. What they mean depends on the depth buffer-format and how far stuff is away from the near plane.
Make sure the both depth format and the projection matrix end up similar in D3D9 and D3D11.

It would be so much easier to find the issue if it was possible to debug D3D9 and D3D11 shaders at the same time, but Visual Studio only supports Direct3D11 for that debugging.
But at the same time, this problem is not really a big issue, since I just in my D3D9 shaders add a higher value than the default one.

When I debugged the spotlight shadow texture, here is the results:

Direct3D9:

Texture (through checking the TexturePtr for the spotlight):
mFormat = PF_FLOAT32_R (32)
mDesiredFormat = PF_FLOAT32_R (32)

Texture view projection matrix (through OgreGpuProgramParams.cpp):
{0.500000000, 0.00000000, 0.555306375, -3.42314982}
{0.500000060, -0.555306256, 5.96046448e-08, -1.86013675}
{1.00133526, 0.00000000, 1.19368451e-07, -2.27511454}
{1.00000012, 0.00000000, 1.19209290e-07, -2.17208099}

Direct3D9 format (through ShadowTextureManager::getShadowTextures -> createInternalResourcesImpl):
d3dPF = D3DFMT_R32F (114)

Direct3D11:

Texture (through checking the TexturePtr for the spotlight):
mFormat = PF_FLOAT32_R (32)
mDesiredFormat = PF_FLOAT32_R (32)

Texture view projection matrix (through OgreGpuProgramParams.cpp):
{0.500000000, 0.00000000, 0.555306375, -3.42314982}
{0.500000060, -0.555306256, 5.96046448e-08, -1.86013675}
{1.00133526, 0.00000000, 1.19368451e-07, -2.27511454}
{1.00000012, 0.00000000, 1.19209290e-07, -2.17208099}

Direct3D11 format (through ShadowTextureManager::getShadowTextures -> createInternalResourcesImpl):
mD3DFormat = DXGI_FORMAT_R32_FLOAT (41)

The specific texture formats (D3DFMT_R32F and DXGI_FORMAT_R32_FLOAT) for the render systems also seem to match up according to this:
https://docs.microsoft.com/en-us/window ... cy-formats

paroj wrote: Wed May 18, 2022 12:32 pm

Maybe you are rendering depth through FFP in D3D9? For depth bias the caster material is important.

No I am using shaders (look at the ShadowCaster in the first post).

rpgplayerrobin
Gnoll
Posts: 617
Joined: Wed Mar 18, 2009 3:03 am
x 353

Re: Shadow bias problem Direct3D9

Post by rpgplayerrobin »

paroj wrote: Wed May 18, 2022 12:34 pm

If you rotate a spotlight until it is upside down, the shadow texture flips. This creates some unwanted bugs.

can you elaborate on that? The shadow matrix adapts with the shadow texture, so the scene orientation should not matter.

On the other hand having coincident up and dir gives you a wrong rotation matrix..

Exactly when it flips (it usually rotates around before it flips as well) the shadow itself gets different in its pixels. Because imagine it aligning completely perfectly towards a box, which means there is no aliasing or anything like that, but then when it flips (and rotates around before it flips) the aliasing is clearly shown in the texture.
So basically, when it gets flipped, it looks pretty strange for a very short duration (until the rotation has ended).

I remember there was some more serious bug that it caused than that, but I don't remember exactly what it was, it was around 3 years ago I had to fix it for some serious reason.

This is also the same bug as I wrote around 2 years ago to you, but I don't remember what the serious bug it had to do with:

Another bug is with lights and their shadows that happens when lights were not in the same orientation as their setDirection function set them in (which made shadows invalid for my spot-lights in some situations), so I fixed it in ShadowRenderer::prepareShadowTextures:

Code: Select all

// Associate main view camera as LOD camera
texCam->setLodCamera(cam);
// set base
if (light->getType() != Light::LT_POINT)
{
	Node* tmpParentSceneNodeROBIN = light->getParentNode();
	if(tmpParentSceneNodeROBIN)
		texCam->setOrientation(tmpParentSceneNodeROBIN->_getDerivedOrientation());
}
if (light->getType() != Light::LT_DIRECTIONAL)
	texCam->setPosition(light->getDerivedPosition());

I have not tried it, but the flipping/rotating might also cause issues with shadow cookies? Or is the matrix actually still correct even though the flipping/rotating happens? Not sure.

paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Shadow bias problem Direct3D9

Post by paroj »

rpgplayerrobin wrote: Wed May 18, 2022 4:17 pm
paroj wrote: Wed May 18, 2022 12:32 pm

depth bias is expressed in depth-buffer units. What they mean depends on the depth buffer-format and how far stuff is away from the near plane.
Make sure the both depth format and the projection matrix end up similar in D3D9 and D3D11.

It would be so much easier to find the issue if it was possible to debug D3D9 and D3D11 shaders at the same time, but Visual Studio only supports Direct3D11 for that debugging.

you might try to debug D3D9 via Vulkan as suggested by dark_sylinc here: viewtopic.php?p=552359#p552359

Another thing that came to my mind is that D3D9 and D3D11 use different compilers (D3D9 still uses D3DX). So maybe your bias value is stored as mediump (16bit) with D3D9. The smallest possible number is 0.000031 then and your bias is truncated to 0.

paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Shadow bias problem Direct3D9

Post by paroj »

rpgplayerrobin wrote: Wed May 18, 2022 4:30 pm

Exactly when it flips (it usually rotates around before it flips as well) the shadow itself gets different in its pixels. Because imagine it aligning completely perfectly towards a box, which means there is no aliasing or anything like that, but then when it flips (and rotates around before it flips) the aliasing is clearly shown in the texture.
So basically, when it gets flipped, it looks pretty strange for a very short duration (until the rotation has ended).

yes, this is to be expected. However your workaround introduces bugs by itself. If we dont check the vectors to be coincident, we get this:

Code: Select all

>>> Ogre.Math.lookRotation(Ogre.Vector3(0, 1, 0), Ogre.Vector3(0, 1, 0))
Matrix3(0, 0, 0; 0, 0, 1; 0, 0, 0)

OgreShadowRenderer.cpp:
From:

this change seems legit. However it would be nice to document why you actually needed it. Its effect is skipping the re-derivation in SceneNode::setDirection.

rpgplayerrobin
Gnoll
Posts: 617
Joined: Wed Mar 18, 2009 3:03 am
x 353

Re: Shadow bias problem Direct3D9

Post by rpgplayerrobin »

yes, this is to be expected. However your workaround introduces bugs by itself. If we dont check the vectors to be coincident, we get this:
Ogre.Math.lookRotation(Ogre.Vector3(0, 1, 0), Ogre.Vector3(0, 1, 0))
Matrix3(0, 0, 0; 0, 0, 1; 0, 0, 0)

I don't understand, how will it introduce bugs? The "lookRotation" line is never used for spotlights in my code.
The orientation has already been set to the cameras orientation as well (before getShadowCamera is called), so it can never go wrong as far as I see it?
With my code, the texture is actually used exactly the same way as the orientation of the light is. I don't see the reason it should be shown differently compared to its orientation.

And yeah, I wish I remember the reason that I needed to fix this bug 3 years ago. I remember it was a huge bug regarding spotlights and that it took days to solve (because I didn't understand Ogre very much back then compared to now).
It could have been that the matrix became invalid compared to the texture itself in that older Ogre version, but that the bug regarding that may have been fixed now.

rpgplayerrobin
Gnoll
Posts: 617
Joined: Wed Mar 18, 2009 3:03 am
x 353

Re: Shadow bias problem Direct3D9

Post by rpgplayerrobin »

Another thing that came to my mind is that D3D9 and D3D11 use different compilers (D3D9 still uses D3DX). So maybe your bias value is stored as mediump (16bit) with D3D9. The smallest possible number is 0.000031 then and your bias is truncated to 0.

In that case I will just keep the special shader code as it is for Direct3D9.

paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Shadow bias problem Direct3D9

Post by paroj »

rpgplayerrobin wrote: Fri May 20, 2022 2:38 pm

The orientation has already been set to the cameras orientation as well (before getShadowCamera is called)

that was the missing puzzle for me. Got it now - upstream change:
https://github.com/OGRECave/ogre/pull/2 ... 6a93f17cf9

Post Reply