Upgrading from 1.11.2 to 13.3.4 Topic is solved

[rpgplayerrobin]

Ogre Version: 13.3.4
Operating System: Windows 10
Render System: Direct3D9

Hey!

I have started upgrading my 1.11.2 project to the 13.3.4 version (downloaded from the first post in https://github.com/OGRECave/ogre/releases), but I have had a lot of problems integrating it with my project.
But the current problem is like hitting a brick wall.

When I try to load Codec_FreeImage_d.dll at startup (from Plugins.cfg like I have always done in my old version), it just does not work.
At the line "mInst = (DYNLIB_HANDLE)DYNLIB_LOAD( name.c_str() );", the mInst becomes NULL, but it works for all other plugins and render systems.
It then throws an error: "Could not load dynamic library .\Codec_FreeImage_d.dll. System Error: The specified module could not be found.".

The .dll file is exactly at the same place as the other .dll files are. And there are no compilation errors regarding importing its .lib file at all.

Then I thought that I could just skip it completely, but then I get an error of "Can not find codec for 'png' format. Supported formats are: astc dds ktx mesh pkm.".
So it seems I really need the freeimage codec.

There are no issues in Ogre.log, it just stops at "22:08:58: Loading library Codec_FreeImage_d.dll".

Since freeimage no longer seems to be a part of Ogre per default, I built it separately (like I am supposed to I guess?) and then just linked it in CMake before I generated the Ogre project.
The compilation of it works and it outputs .dll and .lib files for it in both debug and release.

So... How can I load the freeimage codec correctly?

Also, it seems the codec in debug in the new version (13.3.4) is only 180 KB, but in my old version (1.11.2) it was 5109 KB.
Same goes for the release version, it is 49 KB in the new version and 2095 KB in the old version.
So is something wrong there, and if so, why or how?

[paroj]

sounds like you built freeimage as a dll, but forgot to copy it next to Codec_FreeImage, so the latter cannot be loaded due to missing dll. Unfortunately the error code is still not-found for this case.

Alternatively, you could build freeimage statically for ease of deloyment or use Codec_STBI.

[rpgplayerrobin]

Nice, thank you!
I copied in "FreeImaged.dll" to also be in my directory now and it seems to load the "Codec_FreeImage_d.dll" correctly, but I am now instead using Codec_STBI as you suggested (as it is much lower in file size).

It shows the loading screen with png textures, so I now know it works.

However, I am getting a crash when I am loading a specific texture with it.
Here is the texture: https://ufile.io/wojqkd38
It is only a single channel png, used as A8/L8 (or whatever one channel is) in a few shaders I am using.
There are more textures in my game like this, which are used for various uses (like defining areas on a mesh where it should set its team color, defining where grass in a map is, etc).

So instead of Codec_STBI, I went back to FreeImage instead. But there it instead crashed when loading in a cubic texture (skybox) for some reason, so I am pretty stuck again...

How can I fix this issue?

[paroj]

Here is the texture: https://ufile.io/wojqkd38

works fine with STI here:

Code: Select all

Texture 'whiteAlpha.png': Loading 1 faces(PF_BYTE_LA,4x4x1) with 2 hardware generated mipmaps from Image. Internal format is PF_BYTE_LA,4x4x1.

please provide stack-traces for any crashes you encounter

[rpgplayerrobin]

Hey!

I have looked into it a lot more now and I seem to have a way to reproduce it for you.

I saw that it depends on when I load the texture/material in my startup code, so I created a small code to load it anywhere in the application to see what causes the crash:

Code: Select all

std::string groupName = "hahatest";
Ogre::ResourceGroupManager& tmpResourceGroupManager = ResourceGroupManager::getSingleton();
tmpResourceGroupManager.createResourceGroup(groupName, true);
tmpResourceGroupManager.addResourceLocation("../../../Media/TEST", "FileSystem", groupName);
tmpResourceGroupManager.initialiseResourceGroup(groupName);
tmpResourceGroupManager.loadResourceGroup(groupName);

Then, the directory only contains the texture and its material:

Code: Select all

material testMat
{
	technique
	{
		pass
		{
			texture_unit
			{
				texture whiteAlpha.png
			}
		}
	}
}

So with this code, I moved it from the start of the creation of the window and then just down in the code until it crashed.
Where it in my application crashes is directly after this line:

Code: Select all

TextureManager::getSingleton().setDefaultNumMipmaps(5);

Should I not use that function anymore? Should I just skip it and let it auto generate unlimited mipmaps?
It seems to be a crash bug regarding mipmaps in that case.
I removed that function for my project and the loading now works.

Anyway, here is my stacktrace:

[rpgplayerrobin]

Another question (since the last one is now resolved in my opinion, other than it being a crash bug), how do you handle shadow casters in this newer version?

In the older version, I used this in the material (per pass):

Code: Select all

shadow_caster_vertex_program_ref DepthShadowmapCaster_StaticAlphaRejection_VS
{
}

shadow_caster_fragment_program_ref DepthShadowmapCaster_StaticAlphaRejection_PS
{
}

And this in the code to alter them:

Code: Select all

tmpPass->setShadowCasterVertexProgram("DepthShadowmapCaster_StaticAlphaRejection_VS");
tmpPass->setShadowCasterFragmentProgram("DepthShadowmapCaster_StaticAlphaRejection_PS");

But now it seems those functions in the code does not exist anymore, and that you should instead have a shader caster material.
Therefore, it instead became like this in the material (per technique):

Code: Select all

shadow_caster_material DepthShadowmapCaster_StaticAlphaRejection

And this in the code to alter them:

Code: Select all

tmpTechnique->getShadowCasterMaterial()->getTechnique(0)->getPass(0)->setVertexProgram("DepthShadowmapCaster_StaticAlphaRejection_VS");
tmpTechnique->getShadowCasterMaterial()->getTechnique(0)->getPass(0)->setFragmentProgram("DepthShadowmapCaster_StaticAlphaRejection_PS");

But... Here is my issue.
In the new version, when I alter the vertex/fragment program of a technique from code, it also alters the shadow caster material, which in turn alters it for all other materials that uses that shadow caster material.

In the older version, I dynamically set the shadow caster programs (and their parameters) from code and everything was fine, and I could even clone the mesh materials and alter their version as well.
Now I cannot even clone the material of a mesh and alter it, since it still points to the same shadow caster material. This is a huge issue.

I am not sure how to solve this... I thought about a system that removes the shadow caster material for every material loaded, created and cloned, then replacing it with a cloned version of that shadow caster material.
But even if I got that to work (which would be as huge as creating a whole new sub-project), there is no way I can ever know when to delete those materials, since they don't really know what other materials own them and such.
It would be a huge pool of just shadow materials laying around, never being able to delete them.

How do I solve this? How do other people handle this? Do they never set dynamic parameters to the shadow casters (like alpha rejection values for the shaders, opacity, etc) or change the shader itself of the shadow caster material (to change to GPU animation, etc)?
Or is there something huge that I am missing completely?

[paroj]

Ogre ships fallback code to still support "shadow_caster_vertex_program_ref" in scripts, maybe it can inspire you:
https://github.com/OGRECave/ogre/blob/0 ... .cpp#L2368

Basically, you should have a finite number of caster materials and assign them to normal materials. If you need a different alpha rejection value, same rules as to normal materials apply i.e. if you clone the normal material, also clone the caster.

[paroj]

Should I not use that function anymore?

Thats a bug. On D3D9, Ogre fails to check the upper limit of possible mipmaps. Then 5 are requested, while only 2 are possible.
will be fixed with https://github.com/OGRECave/ogre/pull/2 ... a0f155444a

[rpgplayerrobin]

Ogre ships fallback code to still support "shadow_caster_vertex_program_ref" in scripts, maybe it can inspire you:
https://github.com/OGRECave/ogre/blob/0 ... .cpp#L2368

Basically, you should have a finite number of caster materials and assign them to normal materials. If you need a different alpha rejection value, same rules as to normal materials apply i.e. if you clone the normal material, also clone the caster.

So I should not use "shadow_caster_material X" in my materials, instead I should keep it as it was with "shadow_caster_vertex_program_ref X" and "shadow_caster_fragment_program_ref X"?
There are many warnings that they are depricated, which is worrying me that they will be removed in the future... But lets ignore that right now.
And then in code I should just alter the shadow caster material itself if I want to alter the shadow caster vertex/fragment programs, since it should be unique for each material being loaded this way.

But how do these materials handle the destruction of that new cloned shadow caster material? I don't see anything obvious in the code that they gets destroyed in OgreTechnique.cpp. Are they actually ever destroyed?

Even if in my code I could (somehow, I not even sure how) make all materials that are dynamically created or cloned (which is 99% of them) get their own clone of their shadow caster material, how would I make sure their cloned shadow caster materials are destroyed when the material itself is unloaded/destroyed?

And by the way, there is a crash bug regarding this way of doing it, which makes it impossible in its current state.
Take for example a material which has 4 techniques and one pass in each technique.
Each of the techniques are for a different way of rending it, for example, High, Medium or Low qualities and then one for a special render for some reason.
Then, if you use the "shadow_caster_vertex_program_ref" and "shadow_caster_fragment_program_ref" in at least just two of those passes (when you want it in all 4 to correctly handle their shadows), it will crash since it has already created a material with that name (crash in PassTranslator::translateShadowCasterProgramRef).

I must say that currently it feels like the new system of handling shadow casters are pretty basic or limited, and a step back from the 1.11.2 version of doing it... There must be some way of handling all this easier than it is now, because in the current system a material is not clonable anymore without using special user-code to clone in a correct manner with its shadow caster material and its possible destruction.

[loath]

i create a hash value from the required features (eg. instancing, wind, alpha, UV count, vertex colors, etc). if a material / shader with this hash name exists i reuse it. otherwise i create the necessary resource. same for shadows.

if a feature dynamically changes i recalculate the hash and create / retrieve / assign.

if the level is unloaded i can just free the material / shader resource group:

Code: Select all

Ogre::ResourceGroupManager::getSingleton().clearResourceGroup (group);
Ogre::ResourceGroupManager::getSingleton().initialiseResourceGroup (group);

i do "level streaming" so if a chunk is unloaded i also call "unload" which frees resources that are not referenced by anyone except the resource manager:

Code: Select all

Ogre::ResourceGroupManager::getSingleton ().unloadUnreferencedResourcesInGroup (group);

[paroj]

instead I should keep it as it was with "shadow_caster_vertex_program_ref X" and "shadow_caster_fragment_program_ref X"?

no

I must say that currently it feels like the new system of handling shadow casters are pretty basic or limited, and a step back from the 1.11.2 version of doing it...

the old system would constantly modify/ recompile a material behind the scenes which the new one avoids. I think you must be doing something wrong if you end up with many different caster materials. There should be only a few: e.g. opaque, alpha-test, hw-skinned, wind+alpha-test are the most I can come up with right now. The only case where this might explode is when you start animating alpha-test separately for each caster - but this is not the common case.

in the current system a material is not clonable anymore without using special user-code to clone in a correct manner with its shadow caster material and its possible destruction.

in the new system the caster material is not tied to specific parent but instead is meant to be shared my many (think of opaque or hw-skinning). In that view sharing the caster on clone makes perfectly sense. Again, modifying the caster behavior for each material is neither common nor efficient.

[rpgplayerrobin]

@loath

i create a hash value from the required features (eg. instancing, wind, alpha, UV count, vertex colors, etc). if a material / shader with this hash name exists i reuse it. otherwise i create the necessary resource. same for shadows.

if a feature dynamically changes i recalculate the hash and create / retrieve / assign.

The problem I have is changing their values per object. For static objects that never fade out or anything like that, your system is fine, but it won't work for dynamic objects that needs to change their values many times on run-time.
Then they need a unique shadow caster material for each and every object, which makes that solution impossible.

Also, how do you know your dynamically created shadow caster materials are deleted? Do you do it manually when using mat->clone? Because that clone function does not do it for you, which means you must clone the shadow caster manually and then destroy it manually somewhere.

@paroj

The only case where this might explode is when you start animating alpha-test separately for each caster - but this is not the common case

This is exactly what I do. How else would you fade away the shadow of something that should fade out? All of my dynamically created objects can fade out. Weapons, armors, characters, monsters, etc.
The exact same thing happens for the ZPrePass regarding this, it knows exactly how to fade out the object.
Each shader has a different way of fading out, and that must reflect on the shadows and ZPrePass or everything will be wrong.
So in my opinion it is the common case, and the old system was better in that case, since then you could also choose to use the shadow caster material feature if you really wanted to (which I also do for a couple of stuff).

Again, modifying the caster behavior for each material is neither common nor efficient.

So the new system makes the above behaviour inefficient?
But that should then apply for every real game out there correctly handling fading out of objects.
More inefficient than the old system, or the same? I guess a whole new material for each material in the game is pretty inefficient though, compared to only just setting the shadow caster shader in the same material like it was before.

Is the new version only made for static environments?
How would you solve fading out the shadow of stuff, if many objects in your project would need that?

[paroj]

This is exactly what I do. How else would you fade away the shadow of something that should fade out?

you only use color shadows right? If you are using depth shadows, then you must be using some kind of texture-base dithering as depth shadows are either on or off.
Most applications use depth shadows nowadays, so shadows do not fade with transparency, but just disappear when alpha falls below a threshold.

All of my dynamically created objects can fade out. Weapons, armors, characters, monsters, etc.

assuming you use color shadows, creating separate materials for these is still bad. You should rather be using a single material with ACT_CUSTOM.

More inefficient than the old system, or the same? I guess a whole new material for each material in the game is pretty inefficient though, compared to only just setting the shadow caster shader in the same material like it was before.

the new system is actually more efficient, as it allows caching the material state. The old approach was constantly trashing the caster material while processing the render queue. The new one allows to set-up everything up front, which avoids mid-frame stuttering.

[rpgplayerrobin]

you only use color shadows right? If you are using depth shadows, then you must be using some kind of texture-base dithering as depth shadows are either on or off.
Most applications use depth shadows nowadays, so shadows do not fade with transparency, but just disappear when alpha falls below a threshold.

I am using depth shadows. I am using alpha rejection (with a static texture to make it look nice) to fade the objects away, that is the way I handle the alpha of objects.
Otherwise ZPrePass would not even be possible with fading out of objects either.
Do you really mean most projects just have the shadows either on or off for a whole object? No one handles the objects shadow to fade in/out with the object itself? That would look very poor.
Every project should handle fading out the shadow with the same look that the mesh itself has, this is how it should look and is from my project: https://youtu.be/GR1LKuPxtnQ?t=62

assuming you use color shadows, creating separate materials for these is still bad. You should rather be using a single material with ACT_CUSTOM.

I am not using color shadows, I am using depth shadows.

the new system is actually more efficient, as it allows caching the material state. The old approach was constantly trashing the caster material while processing the render queue. The new one allows to set-up everything up front, which avoids mid-frame stuttering.

But only for materials that never change? Or would it still be more efficient than the last version even if every single material in the game instead used a clone of the shadow caster material?

[paroj]

I am using depth shadows. I am using alpha rejection

yeah, thats a variant of dithering I was referring to:
https://digitalrune.github.io/DigitalRu ... 40ab1e.htm

Every project should handle fading out the shadow with the same look that the mesh itself has, this is how it should look and is from my project:

to be honest, I would not have noticed that the shadows are correct, if not for our discussion here. But that should not stop you from doing it.

I am not using color shadows, I am using depth shadows.

the argument still applies. With ACT_CUSTOM you can supply per-entity values to one common material/ shader.

Or would it still be more efficient than the last version even if every single material in the game instead used a clone of the shadow caster material?

It would certainly waste some RAM - however no CPU cycles would be wasted for validating the material during shadow rendering.

[rpgplayerrobin]

It seems the reality of the situation is that the new version has removed a feature and in no way replaced it with a new feature, which means it has essentially dropped support to fade in/out objects correctly in an advanced system.
The user has to do something about it now instead of the engine doing it like it was in 1.11.2.

Essentially, each material in 1.11.2 was complete. It was 1 step and one file.

Each material in 13.3.4 is incomplete, since it needs another material to function, and you can no longer set shader parameters in the material, which means you need to do that in code instead.
This means it is 1 step for the material, 1 step for the caster material and then 1 step to set the shader parameters in code. 3 different steps for each material instead of 1.

The previous way to define the shadow caster per material:

Code: Select all

shadow_caster_vertex_program_ref DepthShadowmapCaster_StaticAlphaRejection_VS
{
}

shadow_caster_fragment_program_ref DepthShadowmapCaster_StaticAlphaRejection_PS
{
	param_named alphaRejectionGreater float 0.5
}

You could also right then and there set parameters for the shaders, which I did for many materials where I wanted special attributes, such as a different alpha rejection value per material.
This is no longer possible in 13.3.4, which is extremely bad, and I would have to make code at startup to specifically set attributes to these shadow caster materials. That is a major issue, as that means that a material is not even a material anymore as you cannot even define what you need in it anymore when it comes to the shadow caster. In the worst case scenario, you would have to define a whole new shadow caster material per material.

The new way to define the shadow caster per material (which also needs another material to even function):

Code: Select all

shadow_caster_material DepthShadowmapCaster_StaticAlphaRejection

This does not support setting special attributes and is not a real part of the material, which also creates more problems below.
Also, this breaks the standard of normal shaders in passes in materials, as they work exactly like they did in 1.11.2, which is the correct way of handling this feature (as it is just that much more simple).

The previous chain was like this for each object:

• Create the entity from a mesh and attach it to a scene node.
• Clone the material and set it to the entity.
• Fade it in/out.
• On destruction of the object, destroy the cloned material.

The new chain would be like this:

• Create the entity from a mesh and attach it to a scene node.
• Clone the material and set it to the entity.
• NEW! Clone the shadow caster material and set it to the cloned material.
• Fade it in/out.
• NEW! On destruction of the object, destroy the cloned shadow caster material.
• On destruction of the object, destroy the cloned material.

The previous chain was like this for each material (looping through them when they are loaded for each resource group):

• Check if we should alter the current material to something else (to support GPU animation for example, controlled from the options menu, material specific stuff, etc).
• Alter the materials shadow caster shaders to the correct one needed. Also set special parameters to them if needed.
• Cloned materials that use this material then also gets the correct shadow caster shaders, which is perfect.

The new chain would be something like this for each material:

• Check if we should alter the current material to something else (to support GPU animation for example, controlled from the options menu, material specific stuff, etc).
• NEW! ??? Clone the shadow caster material somehow? But place it where? How would I know when the base material gets unloaded? This is impossible. I have asked this question many times in this forum post, which I have not received an answer to, which means this must be done by the user, but its almost impossible (or at least extremely hard) to know how to do that in an advanced system.
• Alter the materials shadow caster shaders to the correct one needed. Also set special parameters to them if needed.
• ALTERED! Cloned materials that use this material does not get the shadow caster shader, that must be done specifically instead by the user somehow, which is also confusing.

Also, exporting materials with the new system in 13.3.4 would be impossible, as the shadow caster material would be cloned with another name and therefore would be impossible to load an exported material.
Then, even more user code would have to be created to just handle this little issue.
In 1.11.2, the material would be exported with the specific shadow caster shader and all its shader attributes instead, and there would be no issue at all. There would be no new name that you would have to relink or something like that, it would just work.

You also talk about ACT_CUSTOM, but that won't work even if it is per object, since the shaders themselves in the shader caster material can change at any given moment, and so can their shader inputs.
It would be a very complex system to have to use renderable->setCustomParameter on every object for each time its shader is altered, and also in a specific manner (since setCustomParameter is index based, and not name based like setNamedConstant). That means for every shader that index changes, and the code would have to somehow keep a track on that for each and every update of the shader.
Or, you would have to just have a simpler project with almost no shaders or features at all.
In 1.11.2 this worked in just one simple function (setNamedConstant) and it handled everything automatically, not even having to know if the shader has the parameter or not or where that parameter exist as an index in the shader.
But even if ACT_CUSTOM somehow got a named function like that, I would still have to handle everything I have talked about even if I used ACT_CUSTOM regarding the shader caster materials, as their shaders needs to be exchanged.
Also, it cannot be set directly in the material definition like the 1.11.2 system could. That has to be taken care through code, which again makes a material not really a material anymore.
ACT_CUSTOM is not flexible since it always has to input a Vector4. Sometimes it might be needed to input stuff like a matrix, and that would also need many user defined functions, which again breaks the pattern of the simple system as it is for all other shaders with named parameters.

I see it as I have a few choices regarding this issue:

1. I do special user code to handle the new shadow caster materials.
   The user code required to handle the per-object shadow caster materials is extremely hard to do though, and I would argue that it is extremely poor design that every user needs to do this, instead of it being supported by the engine, like it used to be in 1.11.2.
   It would clutter the entire project with extra code, double the amount of materials and would most likely create bugs (since if it is missed somewhere, it will be invalid for that object and all future objects using that material).
   And I still don't know how to handle the destruction of the shadow caster materials correctly, as those cloned ones will be set on materials that I never know when they will be destroyed.
   They can either be a material loaded by a resource group, a material created from scratch or a material cloned, which makes it extremely complex to try to handle the destruction of the shadow caster materials. This was all handled automatically in 1.11.2 since it was instead in the actual material and not a required extra material.
   .
2. I remove all code in Ogre for shadows.
   Then I create my own material scheme in every material in the game for it, which goes to the old (better) system of doing stuff, which means it is in the same material instead.
   That would of course mean I would have to handle rendering the shadows myself instead, and it would be extremely hard, time consuming and complex to do correctly for multiple shadows.
   The drawback of this system is that each material would only handle one way of casting shadows, and not per-pass like the system was before, so this is not a good alternative at all compared to the one in 1.11.2.
   .
3. I pray and wait for a future Ogre version that gets back support for this feature, making a material a material again.
   But that seems very unlikely since this forum post shows pretty clearly that it is intended and that Ogre now only targets simpler projects with less advanced features.
   .
4. I stop trying to upgrade, since the upgrade is actually a downgrade, since the new version is worse than the old version when it comes to this dealbreaker. I don't think the new shadow caster system is very thought through, and I would never think that such a simple system as it was before could became this big of a problem.
   This would probably mean the death of my project though, as I would no longer be able to upgrade the engine.
   I would also have to choose another engine for my next project of course, as this is no longer able to be used for bigger projects. This is really depressing to see.

I would suggest to anyone else with an advanced project thinking about upgrading to 13.3.4 to wait until this design error has been fixed.
Otherwise you will have a very bad time when it comes to shadow casters (if you want to handle them correctly that is, with multiple shaders/materials in a dynamic system).

[paroj]

I am getting strong xkcd vibes here: https://xkcd.com/1172/

look, conceptually each entity has two appearances when factoring-in texture shadows

1. one that is rendered on screen
2. one that is rendered into the shadowmap

for most applications 1 >> 2 and often one can even get away with the default caster material for the whole scene. So this is the case I want to be easy and fast.
The rationale of new vs. old system was discussed in some detail here: viewtopic.php?t=36167

Your use-case is quite specific, so while I agree that it must be possible, it is ok that you have to put some effort into it.

I think it translates to the new system reasonably well:

Code: Select all

// some non-trivial sample
material Old
{
        technique
        {
        	scheme Bad
        	...
        }
	technique
	{
		scheme Good
		pass
		{
			shadow_caster_fragment_program_ref Shader
			{
				param_named Foo ...
			}
		}
	}
}

becomes

Code: Select all

// previously this material would be created on the fly, preventing caching
material NewCaster
{
	technique
	{
		pass
		{
			fragment_program_ref Shader
			{
				param_named Foo ...
			}
		}
	}
}

material New
{
        technique
        {
        	scheme Bad
        	...
        }
	technique
	{
		scheme Good
		shadow_caster_material NewCaster
	}
}

the new system correctly reflects that the object has two different appearances. This does not mean the material is incomplete.

You might argue that we already have techniques for this, and you would be right. Someday we can maybe do shadows that way: https://github.com/OGRECave/ogre/issues/892
However, shadow_caster_material exists since 1.7 so we broke less code with it - while achieving the same goal.

You also talk about ACT_CUSTOM, but that won't work even if it is per object, since the shaders themselves in the shader caster material can change at any given moment, and so can their shader inputs.
It would be a very complex system to have to use renderable->setCustomParameter on every object for each time its shader is altered, and also in a specific manner (since setCustomParameter is index based, and not name based like setNamedConstant). That means for every shader that index changes, and the code would have to somehow keep a track on that for each and every update of the shader.

this is not how ACT_CUSTOM works, see https://ogrecave.github.io/ogre/api/lat ... 68a54a3da3

How would I know when the base material gets unloaded? This is impossible. I have asked this question many times in this forum post, which I have not received an answer to, which means this must be done by the user, but its almost impossible (or at least extremely hard) to know how to do that in an advanced system.

You register a Resource::Listener and if you go with a 1:1 mapping for the caster materials, you just unload them unconditionally. Otherwise you check the use_count().

[loath]

hope you get this sorted out @rpgplayerrobin. i've had a great experience with 13.x and my project has it's own quirks. i just watched (and subscribed on youtube) to the video above. your effects are awesome. love the grass, impact effects on the terrain, the enemy shields and ragdolls that go flying, etc.

[rpgplayerrobin]

@loath

hope you get this sorted out @rpgplayerrobin. i've had a great experience with 13.x and my project has it's own quirks. i just watched (and subscribed on youtube) to the video above. your effects are awesome. love the grass, impact effects on the terrain, the enemy shields and ragdolls that go flying, etc.

Thank you!
I will not upgrade, I will have to wait for a future version that fixes the issue, or simply just take what code I need for my current version of Ogre.

@paroj

You might argue that we already have techniques for this, and you would be right. Someday we can maybe do shadows that way: https://github.com/OGRECave/ogre/issues/892

That would then be the fix to this whole issue. I would rather wait for that than even attempt to throw more hours at this issue (since the code is already a huge mess at this stage of not even working correctly).
That will probably remove all needs (if it renders all different shadow textures, and not just one) to try to somehow make a material a material again in the user code and getting a whole lot of spaghetti code to try to make it work.

I am getting strong xkcd vibes here: https://xkcd.com/1172/

Funny, but if it was serious I am pretty sure you did not read through my post correctly in that case, since I there in detail showed logically how poorly designed the new shadow caster system is and how much user code is needed to fix it.
It has nothing to do with a bug that was fixed that I somehow abused, it has only to do with the difficulty of the new 2 materials per material compared to the previous single-material.
Also, it is very hard to compare that post to remaking the system in such a way that more advanced projects gets it extremely hard, that (from my last post as well) shows that Ogre 13.3.4 targets much more simpler projects.
Even if there is a performance increase in the new system, it requires so much work/spaghetti code to be able to make it work again. And also, even if I got it working I am pretty sure it would actually have the same or worse performance compared to before regarding this system, as the amount of material would be doubled if you want a unique shadow caster per material. That also means that looping through and fixing materials would be slower and other functions like "MaterialManager::getByName" (though not by a lot).

look, conceptually each entity has two appearances when factoring-in texture shadows

one that is rendered on screen
one that is rendered into the shadowmap

This has not changed because of this change. The only difference is it needs one more material than before, which is extremely complex on how to handle correctly, detailed in my previous post.

for most applications 1 >> 2 and often one can even get away with the default caster material for the whole scene. So this is the case I want to be easy and fast.
The rationale of new vs. old system was discussed in some detail here: viewtopic.php?t=36167

This is great for very simple applications. Nothing else. No one talks about cloning or special materials in there, since I was apparently the first who has ever brought it up. Everything with that system is designed to be static.

Your use-case is quite specific, so while I agree that it must be possible, it is ok that you have to put some effort into it.

But the effort is not just effort. It is handling all cases I already wrote in detail, which are huge issues.

the new system correctly reflects that the object has two different appearances. This does not mean the material is incomplete.

Actually, it does. Since if you copy the material (clone), it is not a complete copy of it. It only copies the base material and not how its shadow is rendered. That is a huge oversight in the design (even if it is more optimized for very simple applications), and to leave it up to user to fix it is also an oversight.

this is not how ACT_CUSTOM works, see https://ogrecave.github.io/ogre/api/lat ... 68a54a3da3

I also mentioned directly after that even if it would work, it would not take care of my user case, which again, is something everyone using advanced projects should have problems with.

You register a Resource::Listener and if you go with a 1:1 mapping for the caster materials, you just unload them unconditionally. Otherwise you check the use_count().

So in that case all users still needs to clone each shadow caster material (somehow) each time it is loaded, created or cloned. It still does not fix anything. Also, checking use_count like that is a bit strange since it might be referenced in a list somewhere still, and will be messed up with user code. It feels like the only current solution to this is that way you mention, but that is extremely similar to GC.Collect in C# and breaks the C++ way of having full control over creation/destruction.

[paroj]

You might argue that we already have techniques for this, and you would be right. Someday we can maybe do shadows that way: https://github.com/OGRECave/ogre/issues/892

That would then be the fix to this whole issue. I would rather wait for that than even attempt to throw more hours at this issue (since the code is already a huge mess at this stage of not even working correctly).
That will probably remove all needs (if it renders all different shadow textures, and not just one) to try to somehow make a material a material again in the user code and getting a whole lot of spaghetti code to try to make it work.

there is an unresolved issue though: currently you can easily couple a material scheme to a shadow caster material (say both should use HW skinning). Thats why the shadow caster material ref is per technique.
If the shadow caster material would be implemented as a material scheme itself, there would be no such coupling. One would need to take care that the shadow caster scheme is appropriate for the on-screen scheme. Need to think some more about whether that would actually that bad.

since I there in detail showed logically how poorly designed the new shadow caster system is and how much user code is needed to fix it. It has nothing to do with a bug that was fixed that I somehow abused,

you have shown that with the new system, your use-case got harder to implement. I have never doubted that though.

The problem with the old system was that it stored two sets of shaders in a single pass (normal & shadow caster). This increased pass size even if you did not use per-material caster shaders. Due to the duplication only vertex & fragment shaders were supported for casters. So if you did something in a geometry shader, your shadows would not match.
Furthermore, your shadow material would always bind all normal textures, as we cannot know whether you need those normal-map or not.
All of this is solved with the new system. I already mentioned the part where we can cache the rasterstate if the caster pass is separated. This may not be immediately useful for D3D9, but improves performance with Vulkan.

Since if you copy the material (clone), it is not a complete copy of it. It only copies the base material and not how its shadow is rendered. That is a huge oversight in the design (even if it is more optimized for very simple applications), and to leave it up to user to fix it is also an oversight.

so do I read it correctly, that all of you critique boils down to this? That the shadow caster material is not cloned on material clone?
You do realize that changing this would be about 3 lines of code in your local Ogre copy?
I am totally not convinced that this would be a sane default behavior though.

checking use_count like that is a bit strange since it might be referenced in a list somewhere still, and will be messed up with user code. It feels like the only current solution to this is that way you mention, but that is extremely similar to GC.Collect in C# and breaks the C++ way of having full control over creation/destruction.

You can also use unloadUnreferencedResourcesInGroup as loath suggested if you have points where you know that many dead shadow caster materials are left behind. That one is implemented via checking use_count though..

[rpgplayerrobin]

One would need to take care that the shadow caster scheme is appropriate for the on-screen scheme.

Exactly, that is what I also thought about at the #2 solution I wrote about. It is a flawed system but at least better than the current one since it uses only one material per object.
An ugly fix for that would be one shadow scheme per normal scheme:
"High" = "Shadow_High"
"Medium" = "Shadow_Medium"
"Low" = "Shadow_Low"
But of course, that is very ugly.

Furthermore, your shadow material would always bind all normal textures, as we cannot know whether you need those normal-map or not.

Does that means that the shadow material does not in its current state bind any textures from the normal material at all?
If that is the case, do each material needs to define their textures in two places (one for the normal material and one for the shadow caster material)?
And if it binds the textures automatically, how? By texture unit name or by index? And does it automatically then also fetch the new texture in the shadow caster material if it is changed on the base material?

The problem with the old system was that it stored two sets of shaders in a single pass (normal & shadow caster). This increased pass size even if you did not use per-material caster shaders. Due to the duplication only vertex & fragment shaders were supported for casters. So if you did something in a geometry shader, your shadows would not match.
Furthermore, your shadow material would always bind all normal textures, as we cannot know whether you need those normal-map or not.
All of this is solved with the new system. I already mentioned the part where we can cache the rasterstate if the caster pass is separated. This may not be immediately useful for D3D9, but improves performance with Vulkan.

But that does not mean that the new system would have to stop making it work in a dynamic manner.
It could have instead also had the option to be defined in the base material which would have the same kind of clone/load system as before (since this feature was completely removed, which is the whole issue), or just by a material parameter setting if it is an external material or an internal material, which would handle the clone/load system as well as its destruction on unload if it is an internal material.
Then both systems would still be operational exactly as before, except using the shadow caster material instead of the shadow_caster_x_program_ref functions.
The problem is that there is no handle for this currently without making many changes in the source and in the user code.

so do I read it correctly, that all of you critique boils down to this? That the shadow caster material is not cloned on material clone?
You do realize that changing this would be about 3 lines of code in your local Ogre copy?
I am totally not convinced that this would be a sane default behavior though.

I would rather have a dynamic system instead of a static one for sure.
But what you suggest here is that I have to make every loading of a material create a clone of its shadow caster, every "->clone" of a material to clone its shadow caster, and then in the user code make every material created from scratch clone its shadow caster, as well as the function destroying a material also destroying its shadow caster material.
But of course, this is overkill and there is nothing that says this would just be less optimized than the old system because of all the extra materials.
It feels a bit strange for such a static system to be the norm, the system was changed from a dynamically made system and a static system, to only a static system.

And I am not sure if it is fixed already, but the old functions should also probably be removed or fixed (shadow_caster_vertex_program_ref / shadow_caster_fragment_program_ref), since they are just crashing if you use them on two different passes on the same material (crash in PassTranslator::translateShadowCasterProgramRef).

[paroj]

Furthermore, your shadow material would always bind all normal textures, as we cannot know whether you need those normal-map or not.

Does that means that the shadow material does not in its current state bind any textures from the normal material at all?
If that is the case, do each material needs to define their textures in two places (one for the normal material and one for the shadow caster material)?

yes and yes. Before you burst out in rage: I am aware that this requires even more code on your side. Currently all alpha-tested materials require this duplication. However, using shadow caster schemes will not solve this either. So one will have to live with the increased verbosity.

But that does not mean that the new system would have to stop making it work in a dynamic manner.

actually, it had to. Modern GPU APIs like Vulkan like having everything configured upfront. You dont want to create a new material mid-frame while processing the render-queue. However, that was exactly how shadow caster materials used to work. This also made the materials mix-and-match able or dynamic as you call it. I doubt that it was designed to be that. Hence my xkcd reference.

With the explicit shadow caster material, users can configure and share the caster materials upfront. This matches how the underlying APIs work and is a good thing IMO.

But what you suggest here is that I have to make every loading of a material create a clone of its shadow caster, every "->clone" of a material to clone its shadow caster, and then in the user code make every material created from scratch clone its shadow caster, as well as the function destroying a material also destroying its shadow caster material.

cloning the shadow caster on ->clone() should be sufficient. Some materials might not have a custom shadow caster, which means using the global one.
To easily decide whether the shadow caster should be deleted with the material, you can tag it with a suffix to its name or put it in a specific resource group.

And I am not sure if it is fixed already, but the old functions should also probably be removed or fixed (shadow_caster_vertex_program_ref / shadow_caster_fragment_program_ref), since they are just crashing if you use them on two different passes on the same material (crash in PassTranslator::translateShadowCasterProgramRef).

thanks for reminding, on it.

[rpgplayerrobin]

yes and yes. Before you burst out in rage: I am aware that this requires even more code on your side. Currently all alpha-tested materials require this duplication. However, using shadow caster schemes will not solve this either. So one will have to live with the increased verbosity.

No rage! I am simply emphasizing the new systems weaknesses.
But this of course makes only the system even worse than I ever imagined.
There is not much more to be said, I'll mark your answer as the accepted answer and the next project will not be with Ogre unless there are only 100% static objects/not any fade in it (which I very highly doubt).

[paroj]

please let us know which engine you choose for your next project - maybe it will inspire a better solution for the shadow issues.

[rpgplayerrobin]

There are also some changes regarding texture filtering (and a lot of other things, everything that has to do with the new SamplerPtr).

When you clone a material, its Sampler is copied over as a reference, which means that if you change it in your new material it will also change the old material.

To get cloned materials working with this correctly without altering its parent, you need to do this in your user code (this only creates new samplers if they are different than another one, by using a hash):

.h:

Code: Select all

class CSampler
{
public:
	CSampler()
	{
		m_isEmpty = true;
	}

~CSampler()
{
}

SamplerPtr m_textureSampler;
bool m_isEmpty;
};
static std::unordered_map<std::string, CSampler> m_samplers;
static void StartSampler(TextureUnitState* textureUnitState);
static void FinishSampler(TextureUnitState* textureUnitState);
static void DestroySamplers();

.cpp:

Code: Select all

void CGeneric::DestroySamplers()
{
	// Destroy all samplers
	m_samplers.clear();
	TextureManager::getSingleton().removeAllNamedSamplers();
}

void CGeneric::StartSampler(TextureUnitState* textureUnitState)
{
	// Setup the temporary filtering
	CSampler& tmpSampler = m_samplers["TEMPORARY"];
	if (tmpSampler.m_isEmpty)
	{
		tmpSampler.m_isEmpty = false;
		tmpSampler.m_textureSampler = TextureManager::getSingleton().createSampler(CGeneric::GenerateUniqueName());
	}

// Copy the sampler of the original texture unit state
SamplerPtr tmpCurrentSampler = textureUnitState->getSampler();
*tmpSampler.m_textureSampler = *tmpCurrentSampler;

// Set the temporary sampler to the texture unit state
textureUnitState->setSampler(tmpSampler.m_textureSampler);
}

void CGeneric::FinishSampler(TextureUnitState* textureUnitState)
{
	// Get a hash for the wanted filtering from the temporary sampler
	SamplerPtr tmpTemporarySampler = textureUnitState->getSampler();
	std::string tmpHash = CGeneric::ToString(tmpTemporarySampler->getMipmapBias()) + " " +
						  CGeneric::ToString(tmpTemporarySampler->getBorderColour()) + " " + 
						  CGeneric::ToString(tmpTemporarySampler->getAddressingMode().u) + " " + 
						  CGeneric::ToString(tmpTemporarySampler->getAddressingMode().v) + " " + 
						  CGeneric::ToString(tmpTemporarySampler->getAddressingMode().w) + " " + 
						  CGeneric::ToString(tmpTemporarySampler->getFiltering(FT_MIN)) + " " +
						  CGeneric::ToString(tmpTemporarySampler->getFiltering(FT_MAG)) + " " +
						  CGeneric::ToString(tmpTemporarySampler->getFiltering(FT_MIP));

// Check if the sampler has not been created yet for this hash
CSampler& tmpSampler = m_samplers[tmpHash];
if (tmpSampler.m_isEmpty)
{
	// Create the sampler for this hash
	tmpSampler.m_isEmpty = false;
	tmpSampler.m_textureSampler = TextureManager::getSingleton().createSampler(CGeneric::GenerateUniqueName());
	*tmpSampler.m_textureSampler = *tmpTemporarySampler;
}

// Set the new sampler to the texture unit state
textureUnitState->setSampler(tmpSampler.m_textureSampler);
}

Usage when you change stuff:

Code: Select all

CGeneric::StartSampler(tmpTextureUnitState);
tmpTextureUnitState->setTextureAddressingMode(tmpOriginalTextureUnitState->getTextureAddressingMode());
tmpTextureUnitState->setTextureBorderColour(tmpOriginalTextureUnitState->getTextureBorderColour());
tmpTextureUnitState->setTextureFiltering(tmpOriginalTextureUnitState->getTextureFiltering(FilterType::FT_MIN),
	tmpOriginalTextureUnitState->getTextureFiltering(FilterType::FT_MAG),
	tmpOriginalTextureUnitState->getTextureFiltering(FilterType::FT_MIP));
CGeneric::FinishSampler(tmpTextureUnitState);

When you exit the program:

Code: Select all

CGeneric::DestroySamplers();

This is not that bad though, since it is not something that you need to alter each frame, which means they can be hashed like this very easily and be kept to a very limited amount of different samplers (in my case around 10 at most).