Should I avoid RTSS invalidateMaterial? Topic is solved

Problems building or running the engine, queries about how to use features etc.
Post Reply
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Should I avoid RTSS invalidateMaterial?

Post by Oogst »

Ogre Version: 1.12.11
Operating System: Windows 10
Render System: DirectX11

After updating to from Ogre 1.9 to 1.12 I've noticed a bunch of bugs in my game. I've now discovered that the problem is that if I edit a material after first creation, I need to call ShaderGenerator::invalidateMaterial to regenerate the shader. I'm editing the material for example to do a smooth fade, and in some cases I might be editing dozens of materials every frame, or more. So my question is: is invalidateMaterial cheap enough that this is a valid approach at all?

The alternative I can think of is to write my own shaders for these specific objects and not use RTSS for these materials, but that seems a bit lame since RTSS can do the job perfectly well and they're really simple materials. Are there other alternatives? Or is invalidateMaterial fast enough and this okay anyway?

Also, I'm currently not using the RTSS shadercache since my loading times are fast enough as is. Is this cache only relevant for between runs, or does this influence the price of invalidateMaterial as well?
Last edited by Oogst on Fri May 07, 2021 3:08 pm, edited 2 times in total.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by paroj »

Oogst wrote: Fri Apr 23, 2021 3:59 pm Ogre Version: 1.12.11
Operating System: Windows 10
Render System: DirectX11

After updating to from Ogre 1.9 to 1.12 I've noticed a bunch of bugs in my game. I've now discovered that the problem is that if I edit a material after first creation, I need to call ShaderGenerator::invalidateMaterial to regenerate the shader. I'm editing the material for example to do a smooth fade, and in some cases I might be editing dozens of materials every frame, or more. So my question is: is invalidateMaterial cheap enough that this is a valid approach at all?
this depends which codepath you hit. If a new shader needs to be generated & compiled this will stall the whole pipeline and is probably the worst thing you can do.
However, if you e.g. only change the diffuse colour, this will merely update the uniform which is very cheap. In this case invalidateMaterial is not actually necessary and you could just directly update the pass that getBestTechnique gives you.

Anyway, compiling a few dozen shaders per frame should be fine. Also, the RTSS caches the programs, so it can re-use a previously compiled shader.

On the other hand, having too many shader variants slows down the rendering due to shader switches.
This is a more general problem though and heavily depends on how many objects you render:
- using a single shader and control the variants via uniforms is best when only a few objects are rendered per variant
- using many shaders, each for a variant is best when many objects are rendered per variant
you want to reduce the shader-switches/ objects-per-variant ratio so to say.

To find out which strategy the RTSS follows, you will need to look at the individual SRS - e.g.
- changing the fog type, results in a new shader
- changing the alpha test function is just a uniform update
Oogst wrote: Fri Apr 23, 2021 3:59 pm Also, I'm currently not using the RTSS shadercache since my loading times are fast enough as is. Is this cache only relevant for between runs, or does this influence the price of invalidateMaterial as well?
the only valid use of the RTSS shader cache is debugging the RTSS shaders/ monitor the RTSS shader generation - for everything else
https://ogrecave.github.io/ogre/api/lat ... a30461a108
is sufficient.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

Okay, thanks for explaining! In that case I'll just do it as I do now and will reconsider if I actually happen into any performance problems. Sounds like I'll probably be fine, given the relatively low technical complexity of my game's graphics. :)

Thanks for the quick reply!
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

Hmm, it seems there's a bit more to this than I realised previously. On my lightning fast CPU I still get only 70fps in a visually rather simple game. That does not bode well for slower computers. So I threw a profiler at it and it says that 79% of the time is spent in Ogre::RTShader::ShaderGenerator::preFindVisibleObjects().

Next I added some logging to Ogre::TargetRenderState::acquirePrograms(). If I'm reading the Ogre source correctly, once we get there the shader will actually be generated and compiled this frame. There I printed the material names that are causing this:

Code: Select all

Ogre/RTShader/shadow_caster_dq_skinning_1weight
Ogre/RTShader/shadow_caster_dq_skinning_2weight
Ogre/RTShader/shadow_caster_dq_skinning_3weight
Ogre/RTShader/shadow_caster_dq_skinning_4weight
Ogre/RTShader/shadow_caster_dq_skinning_1weight_twophase
Ogre/RTShader/shadow_caster_dq_skinning_2weight_twophase
Ogre/RTShader/shadow_caster_dq_skinning_3weight_twophase
Ogre/RTShader/shadow_caster_dq_skinning_4weight_twophase
Ogre/RTShader/shadow_caster_skinning_1weight
Ogre/RTShader/shadow_caster_skinning_2weight
Ogre/RTShader/shadow_caster_skinning_3weight
Ogre/RTShader/shadow_caster_skinning_4weight
RTSS/PerPixel_SinglePass
RTSS/NormalMapping_SinglePass
RTSS/NormalMapping_MultiPass
RTSS/NormalMapping_MultiPass
RTSS/NormalMapping_MultiPass
RTSS/Athene/NormalMapping_SinglePass
RTSS/Athene/NormalMapping_MultiPass
RTSS/Athene/NormalMapping_MultiPass
RTSS/Athene/NormalMapping_MultiPass
RTSS/NormalMapping_MultiPass_2lights
RTSS/NormalMapping_MultiPass_2lights
RTSS/NormalMapping_MultiPass_2lights
RTSS/LayeredBlending
Ogre/RTShader/TriplanarTexturing
LoadingScreen
Fonts/MenuFont
CurrentNotesMiniVisualiserBackground
CurrentNotesMiniVisualiserTonalityIndicator
Cursor
ChargePoint
CGLogoOverlay
CGLogoOverlayName
CGPlayerHud
CGXboxController
At a glance that seems to be all materials RTSS has seen so far, including the base Ogre materials and my own. It includes materials that were used before (during this session) but not right now, like the LoadingScreen. So my impression is that RTSS is regenerating all shaders it knows every frame. Is that analysis correct, or is Ogre::TargetRenderState::acquirePrograms() not at that point yet?

I had some calls to Ogre::RTShader::ShaderGenerator::invalidateMaterial() in my code, so for further testing I removed those for the moment. That does not change anything. So it seems like even if I don't invalidate anything, RTSS still regenerates all of these materials every frame. I think?

Any idea what I'm doing wrong here? Why is RTSS eating so much performance?
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by paroj »

materials also get invalidated if their scheme is invalidated. This is triggered in ShaderGenerator::preFindVisibleObjects via synchronizeWithLightSettings and synchronizeWithFogSettings.
So, for instance, if a light starts/ stops affecting the frustum the shaders get recompiled for the actual light count.

Alternatively, you can set a fixed light number for the TargetRenderState via:
https://ogrecave.github.io/ogre/api/lat ... ight_count
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

Most of these materials are set to lighting off, does the number of lights still matter then?

Also, most of these materials aren't actually being rendered in those frames, they were rendered before (like for example the loadingscreen material, which isn't rendered during gameplay). So dynamic light counts shouldn't influence those I suppose?
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by paroj »

Oogst wrote: Fri May 07, 2021 7:38 pm Most of these materials are set to lighting off, does the number of lights still matter then?
no
Oogst wrote: Fri May 07, 2021 7:38 pm Also, most of these materials aren't actually being rendered in those frames, they were rendered before (like for example the loadingscreen material, which isn't rendered during gameplay). So dynamic light counts shouldn't influence those I suppose?
the RTSS does not actually check what is on screen right now. After the material was registered with "createShaderBasedTechnique", it assumes it could be.

I also just checked the code, and even materials with "lighting off" get invalidated.

On the other hand, the RTSS caches shaders, so if the same shader code is being generated, it will not hit the D3D compiler.

But all in all, the RTSS is currently not particularly clever on how to avoid shader re-building.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

Hmm, given how much performance this costs, it sounds like my best option then is to just disable RTSS altogether and write my own simple shaders instead.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

I've tried changing all my materials to use shaders instead of RTSS, which increased framerate by 40%. But even then RTSS uses a lot of performance on materials like "Ogre/RTShader/shadow_caster_dq_skinning_1weight". So I've removed RTSS from my code altogether now and now my framerate is 6x higher (from 50fps to 300fps on the scene I tested with).

I don't know whether I was doing something wrong (I'm not using OgreBites so maybe there's something there that I missed) or whether RTSS had a bug, but it does seem problematic that it eats so much performance even when I'm not invalidating any materials.

One thing remains from RTSS: Ogre won't start if the DLL isn't present. I'm not including RTSS or its lib file anymore. How can I keep Ogre from trying to load OgreRTShaderSystem.dll?

By the way, I case anyone ever wants to use Ogre without RTSS: it requires setting font shaders. I found code for how to set font shaders on the Ogre forum elsewhere and modified it for this specific case. It looks like this:

Code: Select all

// Normally fonts are loaded by Ogre automatically, but we need to set their shaders before they are used for the first time.
// To be able to set the shaders before use, we need to load the fonts manually on startup first, and then we can set the shaders.
auto& materialManager = Ogre::MaterialManager::getSingleton();
auto fontIter = Ogre::FontManager::getSingleton().getResourceIterator();
for (auto fontResource : fontIter)
{
	const auto& fontName = fontResource.second->getName();
	fontResource.second->load();
	auto material = materialManager.getByName("Fonts/" + fontName);
	auto pass = material->getTechnique(0)->getPass(0);
	pass->setVertexProgram("FontVP");
	pass->setFragmentProgram("FontFP");
}
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by paroj »

Oogst wrote: Mon May 10, 2021 11:23 am I've tried changing all my materials to use shaders instead of RTSS, which increased framerate by 40%. But even then RTSS uses a lot of performance on materials like "Ogre/RTShader/shadow_caster_dq_skinning_1weight". So I've removed RTSS from my code altogether now and now my framerate is 6x higher (from 50fps to 300fps on the scene I tested with).
could you share that scene or a reproducer for the performance issues you are experiencing? That way I could look into optimizing the RTSS for that your use-case. I guess you are switching lights every frame or similar.
Oogst wrote: Mon May 10, 2021 11:23 am One thing remains from RTSS: Ogre won't start if the DLL isn't present. I'm not including RTSS or its lib file anymore. How can I keep Ogre from trying to load OgreRTShaderSystem.dll?
OgreBites depends on the RTSS, so if you still use that, you also need the RTSS lib around. You can use "Dependency Walker" on Windows to reveal those.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

paroj wrote: Mon May 10, 2021 11:45 amcould you share that scene or a reproducer for the performance issues you are experiencing? That way I could look into optimizing the RTSS for that your use-case. I guess you are switching lights every frame or similar.
Unfortunately it's a rather big project and I can't easily break off a repro case.
paroj wrote: Mon May 10, 2021 11:45 amOgreBites depends on the RTSS, so if you still use that, you also need the RTSS lib around. You can use "Dependency Walker" on Windows to reveal those.
Ah, it might be the OgreBites ConfigDialog then. I think that's the only bit of OgreBites that I'm using, but that one might trigger the attempt at loading OgreRTShaderSystem.dll.

Again, thanks for all the help! You really are a lifesaver, paroj!
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by paroj »

Oogst wrote: Mon May 10, 2021 3:27 pm
paroj wrote: Mon May 10, 2021 11:45 amcould you share that scene or a reproducer for the performance issues you are experiencing? That way I could look into optimizing the RTSS for that your use-case. I guess you are switching lights every frame or similar.
Unfortunately it's a rather big project and I can't easily break off a repro case.
hmm.. could you comment out the "invalidate();" call in "ShaderGenerator::SGScheme::synchronizeWithLightSettings()" & "ShaderGenerator::SGScheme::synchronizeWithFogSettings()" and check whether this resolves your performance issues? This would verify whether I am on the right track with this.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

If I remove those two invalidate() lines then RTSS indeed stops killing my performance and it doesn't come into TargetRenderState::acquirePrograms anymore, unless I invalidate a material by hand or use a material for the first time (as is expected). So yeah, that does seem to fix the particular situation I was encountering. :)

Note by the way that my scene doesn't contain fog and I think the lights also don't move. And regardless of whether they move, the materials that are constantly regenerated have the lighting off property.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by paroj »

yes, the issue is that currently the whole scheme (=all materials used so far) is invalidated, when a single light gets out of view. Just generating all the shaders is costly - even when they end up being ultimately cached.

I guess you have many small spotlights, which would heavily trigger this. In contrast most samples (and Ogre users I guess) use large pointlights/ directional lighting.

as a workaround you can already do:

Code: Select all

// this is fine if everything is unlit
mShaderGenerator->getRenderState(RTShader::ShaderGenerator::DEFAULT_SCHEME_NAME)->setLightCountAutoUpdate(false);
// or additionally set a upper bound of lights as
... ->setLightCount({point, directional, spot});
meanwhile I try to teach the RTSS that it can leave unlit materials alone. Maybe also keep the shader if the light count just decreased smaller than a threshold.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

paroj wrote: Tue May 11, 2021 7:42 pm yes, the issue is that currently the whole scheme (=all materials used so far) is invalidated, when a single light gets out of view. Just generating all the shaders is costly - even when they end up being ultimately cached.

I guess you have many small spotlights, which would heavily trigger this. In contrast most samples (and Ogre users I guess) use large pointlights/ directional lighting.
[...]
I think something other than that might be triggering this problem. I have only 14 lights in my scene, and the problem happens even when the camera and the lights are all static.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by paroj »

I implemented the invalidation strategy I mentioned above and added logging in case invalidation happens.
Maybe you can try this for your scene:
https://github.com/OGRECave/ogre/commit ... 16b2341c4d
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Should I avoid RTSS invalidateMaterial?

Post by Oogst »

I gave it a try and this indeed fixes the issue with RTSS for me. :D

I don't quite understand why though, because I don't think I move, create or edit any lights after loading.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
Post Reply