Page 1 of 1

[SOLVED] Hardware animation & shadows

Posted: Thu Jul 12, 2018 1:41 am
by rpgplayerrobin
Ogre Version: 1.11.2
Operating System: Windows 10
Render System: Direct3D9 (but same thing happens on my Direct3D11 shader)


I am almost complete in upgrading from Ogre 1.9 to Ogre 1.11.2, but I have yet again ran into a problem. :D

Shadow receiver materials are behaving strange when using hardware animation, when not using hardware animation, it works perfectly.
For some reason, when I use the exact same shader/material for hardware animation on Ogre 1.9, it works.
The shadow casting work either way, it is only the shadow receiver that does not work properly.

Below is a very simple shader/material that works on Ogre 1.9 but not in Ogre 1.11.2.


Code: Select all

material alien
			vertex_program_ref TestSkinning_VS

			fragment_program_ref TestSkinning_PS

			texture_unit DiffuseMap
				texture alien.png

			texture_unit ShadowMap
				content_type shadow
				tex_address_mode border
				tex_border_colour 1 1 1 1

Code: Select all

vertex_program TestSkinning_VS hlsl
	source test.hlsl
	entry_point main_vs
	target vs_3_0
	includes_skeletal_animation true
	column_major_matrices false

		param_named_auto worldMatrix world_matrix
		param_named_auto texViewProj texture_viewproj_matrix

		param_named_auto viewProjectionMatrix viewproj_matrix
		param_named_auto inverseWorldMatrix inverse_world_matrix
		param_named_auto worldMatrix3x4Array world_matrix_array_3x4

fragment_program TestSkinning_PS hlsl
	source test.hlsl
	entry_point main_ps
	target ps_3_0

		param_named_auto shadowColour shadow_colour

Code: Select all

float4x4 worldMatrix;
float4x4 texViewProj;

float4x4 viewProjectionMatrix;
float4x4 inverseWorldMatrix;
float3x4 worldMatrix3x4Array[23];

struct VS_OUTPUT
	float4 oVertexPos : TEXCOORD0;
	float2 oUV : TEXCOORD1;
	float3 oShadowUV : TEXCOORD2;

VS_OUTPUT main_vs( float4 position : POSITION,
				   out float4 oPosition : POSITION,
				   float2 iUV : TEXCOORD0,
				   float4 blendIdx : BLENDINDICES,
				   float4 blendWgt : BLENDWEIGHT )

	oPosition = float4(0, 0, 0, 0);
	for(int i = 0; i < 3; ++i)
		oPosition += float4(mul(worldMatrix3x4Array[blendIdx[i]], position).xyz, 1.0) * blendWgt[i];
	Out.oVertexPos = mul(inverseWorldMatrix, oPosition);
	oPosition = mul(viewProjectionMatrix, oPosition);

	Out.oUV = iUV;

	float4 shadowUV = mul(texViewProj, mul(worldMatrix, Out.oVertexPos));
	Out.oShadowUV = / shadowUV.w;

	return Out;

float4 shadowColour;

sampler DiffuseMap : register(s0);
sampler ShadowMap : register(s1);

float4 main_ps( float4 position : TEXCOORD0,
				float2 uv : TEXCOORD1,
				float3 shadowUV : TEXCOORD2 ) : COLOR0
	float4 color = float4(1,1,1,1); = tex2D(DiffuseMap, uv).xyz;

	const float shadowBias = 0.000005;
	float shadowSample = tex2D(ShadowMap, shadowUV.xy).x;
	if(shadowSample + shadowBias < shadowUV.z) *=;

	return color;
When using Ogre 1.11.2, the shadow receiver on all hardware animation materials think they got shadowed when I move the main camera into a shadowed area. Something is very strange here...

I have tried everything I could think of, but nothing fixed it. I tried changing "worldMatrix" to many other types of "param_named_auto", but nothing works correctly.
With some of my tests, I have seen that the shader actually receives the shadow correctly, but that it is for some reason in the wrong space or something, which is wierd because it worked in Ogre 1.9...

Anyone have any ideas on how to fix this?

Re: Hardware animation & shadows

Posted: Thu Jul 12, 2018 1:44 pm
by paroj
probably the aabb of the object is wrong/ out of date with hardware animation as the data is on the GPU only. Try enabling shadowBuffers for that mesh. Also compare Mesh::_computeBoneBoundingRadius with and without hardware animations.

Re: Hardware animation & shadows

Posted: Thu Jul 12, 2018 3:01 pm
by rpgplayerrobin
Thank you for your reply! :)

It seems that "mVertexBufferShadowBuffer" and "mIndexBufferShadowBuffer" are actually true from startup on that mesh, for both with or without GPU animation.

Mesh::_computeBoneBoundingRadius calculates "mBoneBoundingRadius" to the same value for both with or without GPU animation: 0.149066687.

Making the AABB visible shows me that they are correct for both with or without GPU animation (since their scene nodes still actually change position/orientation, and that the entity still has a base AABB).

I would think this has something to do with shaders, since the shadow texture is actually added to the shader but is just visually incorrect.
Why would my GPU animated meshes become fully shadowed when I move my camera into a shadowed area?
Keep in mind that this worked on Ogre 1.9, so it is not a general error, probably just Ogre 1.11.2 error.

Re: Hardware animation & shadows

Posted: Fri Jul 13, 2018 12:29 am
by paroj
hmm.. there are many changes between 1.9 and 1.11 - notably the 1.10 release. Which ShadowCameraSetup are you using? Ideally I would need a minimal reproducer scene to investigate..

Re: Hardware animation & shadows

Posted: Fri Jul 13, 2018 11:41 am
by rpgplayerrobin
Thank you so much to be willing to investigate! :D
I would not be able to fix this myself...

I have made a minimal scene with minimal shaders that shows exactly my problem:


Code: Select all

#ifndef _MAIN_H_
#define _MAIN_H_

using namespace Ogre;

class CModel
	CModel(std::string meshName, std::string materialName, SceneManager* sceneManager)
		m_ent = sceneManager->createEntity(meshName);
		static int idCreator = 0;
		m_node = sceneManager->getRootSceneNode()->createChildSceneNode("CModelSceneNode" + std::to_string(idCreator++));

		if (m_ent->hasSkeleton() && m_ent->getAllAnimationStates()->hasAnimationState("Up_Walk_Forward"))
			AnimationState* tmpAnimationState = m_ent->getAllAnimationStates()->getAnimationState("Up_Walk_Forward");

	void Update()
		if (m_ent->hasSkeleton() && m_ent->getAllAnimationStates()->hasAnimationState("Up_Walk_Forward"))
			AnimationState* tmpAnimationState = m_ent->getAllAnimationStates()->getAnimationState("Up_Walk_Forward");

	Entity* m_ent;
	SceneNode* m_node;

class CApplication
	virtual ~CApplication();

	void Initialize();

	Root* m_Root;
	Camera* m_Camera;
	SceneManager* m_SceneManager;
	RenderWindow* m_Window;
	RenderSystem* m_RenderSystem;

	float m_DT;
	float m_dtNoPause;

	double m_totalApplicationUpTime;

	bool m_skipWindowEvents;
	bool m_skipWindowMovedEvents;
	bool m_skipWindowResizedEvents;

	std::vector<CModel*> m_models;

	void Run();


Code: Select all

#include "StdAfx.h"

int WINAPI WinMain(HINSTANCE hInstance, HINSTANCE hPrevInstance, LPSTR lpCmdLine, int iShowCmd)
	// Create the application object
	CApplication* app = new CApplication();

		// Initialize application

		// Run the application

		// Destroy the application
		delete app;
	catch( Ogre::Exception &e )
		MessageBox( NULL, e.getFullDescription().c_str(), "An ogre exception has occured!", MB_OK | MB_ICONERROR | MB_TASKMODAL);
		return 1;

	return 0;

// Constructor for the CApplication

// Destructor for the CApplication

// Initializes the application
void CApplication::Initialize()
	// Create the root object
	m_Root = new Root("Plugins.cfg", "");

	m_RenderSystem = m_Root->getRenderSystemByName("Direct3D9 Rendering Subsystem");

	m_RenderSystem->setConfigOption("Allow NVPerfHUD", "No");
	m_RenderSystem->setConfigOption("Floating-point mode", "Consistent");
	m_RenderSystem->setConfigOption("Full Screen", "No");
	m_RenderSystem->setConfigOption("sRGB Gamma Conversion", "No");

	// Create the window
	std::string tmpApplicationName = "Test";
	m_Window = m_Root->initialise(false, tmpApplicationName);
	Ogre::NameValuePairList tmpMiscParams;
	tmpMiscParams["monitorIndex"] = "0";
	tmpMiscParams["vsync"] = "0";
	tmpMiscParams["FSAA"] = "0";
	const int tmpNative = 2;
	m_Window = m_Root->createRenderWindow(tmpApplicationName,


	// Create the scene manager
	m_SceneManager = m_Root->createSceneManager("OctreeSceneManager", "testSceneManager");

	// Set the objects to move instead of the camera, have the camera at origin at all times

	// Set the shadow directional light extrusion distance (removes shadow jittering)

	// Create the camera
	m_Camera = m_SceneManager->createCamera("testCamera");
	SceneNode* tmpParent = m_SceneManager->getRootSceneNode()->createChildSceneNode();

	float distance = 2.5f;
	tmpParent->setPosition(Vector3(-distance, 3.5f, -distance));


	m_Camera->setAspectRatio(float(m_Window->getWidth()) / float(m_Window->getHeight()));

	// Create a viewport
	Viewport* tmpViewport = m_Window->addViewport(m_Camera);
	tmpViewport->setBackgroundColour(ColourValue(0, 0, 0));

	// Set default mipmap level

	// Create our resource group
	std::string tmpResourceGroupName = "testResourceGroup";
	Ogre::ResourceGroupManager& tmpResourceGroupManager = ResourceGroupManager::getSingleton();
	tmpResourceGroupManager.createResourceGroup(tmpResourceGroupName, true);

	std::string tmpMediaDirectory = "../../../Media";
	tmpResourceGroupManager.addResourceLocation(tmpMediaDirectory + "/step1", "FileSystem", tmpResourceGroupName);
	tmpResourceGroupManager.addResourceLocation(tmpMediaDirectory + "/step2", "FileSystem", tmpResourceGroupName);


	tmpResourceGroupManager.loadResourceGroup(tmpResourceGroupName, true, true);

	// Create the light that casts shadow
	Light* tmpLight = m_SceneManager->createLight();
	tmpParent = m_SceneManager->getRootSceneNode()->createChildSceneNode();
	tmpParent->setDirection(Vector3(-1.0f, -2.0f, -1.0f).normalisedCopy(), Node::TS_WORLD);

	m_SceneManager->setShadowColour(ColourValue(0.1f, 0.1f, 0.1f, 1.0f));

	// Floor on the ground
	CModel* tmpModel = new CModel("floor.mesh", "floor", m_SceneManager);
	tmpModel->m_node->setScale(30.0f, 30.0f, 30.0f);

	// Two models, one with GPU animation and one without
	tmpModel = new CModel("alien.mesh", "alien_withoutGPUAnimation", m_SceneManager);
	tmpModel->m_node->setPosition(-1.0f, 0.0f, 0.0f);

	tmpModel = new CModel("alien.mesh", "alien", m_SceneManager);
	tmpModel->m_node->setPosition(1.0f, 0.0f, 0.0f);

	// Wall that casts shadows on the two models above
	tmpModel = new CModel("floor.mesh", "floor", m_SceneManager);
	tmpModel->m_node->setScale(3.0f, 10.0f, 3.0f);
	tmpModel->m_node->setPosition(-1.0f, 0.0f, 1.0f);

	// Set the shadow values
	std::string tmpCasterMaterialName = "DepthShadowmapCaster";
	MaterialPtr tmpCasterMaterial = MaterialManager::getSingleton().getByName(tmpCasterMaterialName);

	if (tmpCasterMaterial)

void CApplication::Run()
	while (!m_Window->isClosed())

		// Update the models
		for (size_t i = 0; i < m_models.size(); i++)

		// Update the movement of the wall
		static bool tmpMovingForward = false;
		static float tmpCurrentTranslate = 0.0f;
		if (tmpMovingForward)
			tmpCurrentTranslate += 0.05f;
			if (tmpCurrentTranslate > 5.0f)
				tmpMovingForward = false;
			tmpCurrentTranslate -= 0.05f;
			if (tmpCurrentTranslate < -4.0f)
				tmpMovingForward = true;
		Vector3 tmpPosition = m_models[m_models.size() - 1]->m_node->_getDerivedPosition();
		tmpPosition.x = tmpCurrentTranslate;
		m_models[m_models.size() - 1]->m_node->setPosition(tmpPosition);

		// Update the message pump
		MSG msg;
		while (PeekMessage(&msg, NULL, 0U, 0U, PM_REMOVE))

Media resources download link (.zip, but I could upload it as something else if you need):

Search for "tmpMediaDirectory" in Main.cpp and make sure it is to the path of the local resources I uploaded above.

Here is how it should look if you get it running:

Re: Hardware animation & shadows

Posted: Fri Jul 13, 2018 11:43 am
by rpgplayerrobin
paroj wrote:
Fri Jul 13, 2018 12:29 am
Which ShadowCameraSetup are you using?
I am using the default one, I never set it anywhere in my code.

Re: Hardware animation & shadows

Posted: Fri Jul 13, 2018 2:49 pm
by paroj
thanks for then reproducer. it will take me a few days until i can give it a spin though as i am away over the weekend. one thing i noticed is that you use camera relative rendering. have you already tried disabling that?

Re: Hardware animation & shadows

Posted: Fri Jul 13, 2018 6:25 pm
by rpgplayerrobin
Yeah, setting it to "false" still does not solve it.

Though it does change the appearance of the shadow by a bit, the hardware animated objects get fully shadowed if the origo of the scene (0,0,0) is shadowed. They either get fully shadowed or not shadowed at all when this happens.

This relative rendering setting explains why the hardware animated objects become shadowed when the camera is in a shadowed area, but still does not change the bug.

No worries if it takes a couple of days, I am just glad to get some help! :)

Re: Hardware animation & shadows

Posted: Fri Jul 13, 2018 7:39 pm
by rpgplayerrobin
I finally managed to get a hlsl debugger working properly, and I think I see the bug now. :D

As I am using "float3x4 worldMatrix3x4Array[23];" AND "float4x4 worldMatrix;" it all goes wrong in the core.

See the function "SceneManager::setWorldTransform", there is populates the world matrices.
If it has hardware animation enabled when it goes into "SubEntity::getWorldTransforms", it gets all those bones as float3x4 matrices.
If it does not use hardware animation, it gets the world matrix as a float4x4.

Then in "GpuProgramParameters::_updateAutoParams" it gets the cached (non-dirty) matrices as float3x4, which is correct for bones (world_matrix_array_3x4 aka ACT_WORLD_MATRIX_ARRAY_3x4).
But this also means that the world matrix (world_matrix aka ACT_WORLD_MATRIX) gets its value as a float3x4, as it uses that cached versions first index (as seen in AutoParamDataSource::getWorldMatrix).

Then I use that world matrix to calculate the shadow uv, which gets messed up completely.

How do we solve this bug? I am not sure what to do here...
I now understand the bug completely, but I am not sure how to fix it in an elegant and optimized manner.

Re: Hardware animation & shadows

Posted: Fri Jul 13, 2018 8:17 pm
by rpgplayerrobin
A temporary fix can be used as this (and it actually works), though I am not sure that this is a good solution at all (or optimized):


Code: Select all

float4 shadowUV = mul(texViewProj, mul(worldMatrix, Out.oVertexPos));

Code: Select all

float4x4 worldMatrixFixed = float4x4(
	float4(0, 0, 0, 1)

float4 shadowUV = mul(texViewProj, mul(worldMatrixFixed, Out.oVertexPos));
What is your input on this? Should we fix it in the code or should I just use this shader "hack"?

Re: Hardware animation & shadows

Posted: Sun Jul 15, 2018 2:43 pm
by paroj
nice. thanks for the throughout investigation. I could reproduce the bug without using the demo scene. The root cause is the use of uninitialized memory, which should be fixed with these changes: ... ab8d156cd4

The bug was introduced in 1.11 with the switch to the Affine3 class.

Re: Hardware animation & shadows

Posted: Sun Jul 15, 2018 4:05 pm
by rpgplayerrobin
Thank you so much!
It works now without the shader hack. :D
I'll mark this as solved.