More graceful invalid shader handling + fallback shaders

What it says on the tin: a place to discuss proposed new features.
User avatar
haloman30
Halfling
Posts: 43
Joined: Mon Aug 29, 2022 2:53 pm
x 4

More graceful invalid shader handling + fallback shaders

Post by haloman30 »

Hello!

A fairly straightforward suggestion (though may or may not be so to implement), would be nice to have a way to have more graceful handling of invalid shaders. In my use case, I'm using Ogre-Next as the renderer a general-purpose game engine - which entails users being able to author their own materials and in some cases custom shader code. Currently, invalid shaders result in assertions being thrown - and I've run into this both with implementing custom shader code support but also ran into either asserts or exceptions (I forget which, it's been a while) when datablock parameters are out of range in some cases.

I've managed to work around the invalid shader issue in my own project by inserting my own check at the render system level, which works fine - but would be nice to have some more graceful handling of invalid/missing/etc shaders/materials officially. Similarly, would also be nice to allow for specifying some sort of fallback shader/material in cases where those shaders are missing - but I can imagine that being a bit more involved than adding a bool somewhere.

I do recognize that this (besides the fallback shader stuff) would generally only apply to debug builds, as assertions don't get hit in release builds - so I can understand if that's just a deliberate design decision, but would be super nice to be able to provide a fallback material/shader.

rpgplayerrobin
Bugbear
Posts: 805
Joined: Wed Mar 18, 2009 3:03 am
x 467

Re: More graceful invalid shader handling + fallback shaders

Post by rpgplayerrobin »

This is true even in the non-Next Ogre version.

I have implemented a way to reload shaders during runtime to debug and test stuff, and every time I write something just a tiny little bit wrong, an error is thrown and the application exists.
But, I have not dedicated any time to solve this, since for me it is not such a big issue.

If you find a way to handle this, let us know! :D

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5551
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1402

Re: More graceful invalid shader handling + fallback shaders

Post by dark_sylinc »

You could wrap renderOneFrame() in a try() catch; but it may not always work alright as OgreNext may be left in an inconsistent state.

But a true solution might be easier to implement than we think:

In OgreNext 4.0 (current master branch), Eugene added the concept of "deadline" to shaders: When shaders take too long to compile, the HLMS_CACHE_FLAGS_COMPILATION_REQUIRED flag is set, mCompilationIncompleteCounter is increased, and drawcalls with an uncompiled PSO will not be rendered.

The PSO compilation is deferred to the next frame. This was originally written to fight ANRs (Application Not Responding) on Android.
Right now this feature is Vulkan-only (i.e. VulkanRenderSystem::_render intentionally skips the drawcall if mPso is a nullptr).

It should be relatively trivial to add a feature to reuse this functionality to skip rendering when the shader could not be compiled (if intended by the application because the shader author is user-controlled), instead of throwing errors and asserting. Probably a bit tiresome to find every line that wants to throw or assert, and change it to account for shaders that are allowed to have compiler errors.

rpgplayerrobin
Bugbear
Posts: 805
Joined: Wed Mar 18, 2009 3:03 am
x 467

Re: More graceful invalid shader handling + fallback shaders

Post by rpgplayerrobin »

If it was detected that a compile error happened, the same "shader object" could just be compiled as a very simple working shader instead, like a shader that is just completely red in the pixel shader, to show the user that it failed.
It could also be done to be set in the code for the programmer with a function, such as "renderSystem->setFallbackPixelShader/setFallbackVertexShader".

A warning could be shown in the Ogre.log to something like "X shader failed to compile, used a minimal shader instead" + the compile error, which the programmer could easily catch by using just a listener on the log class (which already exists).

Then, when fixing the error in the shader and reloading it and if it succeeds, it would replace the minimal red shader.
That would make the programmer being able to catch the error from the log and show to the user in any way, and it would also make the user not being able to crash the application.

paroj
OGRE Team Member
OGRE Team Member
Posts: 2283
Joined: Sun Mar 30, 2014 2:51 pm
x 1246

Re: More graceful invalid shader handling + fallback shaders

Post by paroj »

if you reload the material after breaking the shader you will get:

Code: Select all

Warning: material DamagedHelmet has no supportable Techniques and will be blank. Explanation: 
Pass 0: fragment program glTF2/PBR_fs cannot be used - compile error

and the "BaseWhite" material will be used as a fallback.
(Ogre1)

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5551
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1402

Re: More graceful invalid shader handling + fallback shaders

Post by dark_sylinc »

rpgplayerrobin wrote: Tue Aug 26, 2025 4:31 pm

If it was detected that a compile error happened, the same "shader object" could just be compiled as a very simple working shader instead, like a shader that is just completely red in the pixel shader, to show the user that it failed.
It could also be done to be set in the code for the programmer with a function, such as "renderSystem->setFallbackPixelShader/setFallbackVertexShader".

A warning could be shown in the Ogre.log to something like "X shader failed to compile, used a minimal shader instead" + the compile error, which the programmer could easily catch by using just a listener on the log class (which already exists).

Then, when fixing the error in the shader and reloading it and if it succeeds, it would replace the minimal red shader.
That would make the programmer being able to catch the error from the log and show to the user in any way, and it would also make the user not being able to crash the application.

This is nicer but harder to implement. There's a lot of details to it (what if the mesh has a non standard vertex format, what if it's a special pixel shader pass like a GBuffer?), while just skipping rendering of the job is much easier (most of the code to handle such case is already there).

The failure can be notified in other words (like text on screen that doesn't go away until it's handled).

paroj wrote: Tue Aug 26, 2025 8:44 pm

if you reload the material after breaking the shader you will get:

Code: Select all

Warning: material DamagedHelmet has no supportable Techniques and will be blank. Explanation: 
Pass 0: fragment program glTF2/PBR_fs cannot be used - compile error

and the "BaseWhite" material will be used as a fallback.
(Ogre1)

That would be what OgreNext calls low level materials, and OgreNext should be handling fallback on those properly.

The problem here is with Hlms customizations which, although very powerful, become problematic when the customization has a syntax error as it may bring part or the entire Hlms system down and thus a fallback cannot be generated (this is akin to customizing RTSS in Ogre1). Even if the Hlms system was not affected, generating a fallback is still problematic if the object is non-standard at all (e.g. has no positions, or has no UVs but has diffuse maps because the custom vertex shader was supposed to generate this procedurally or sourcing it from a different place like an UAV buffer).

User avatar
haloman30
Halfling
Posts: 43
Joined: Mon Aug 29, 2022 2:53 pm
x 4

Re: More graceful invalid shader handling + fallback shaders

Post by haloman30 »

Would it be easier/simpler to have some mechanism to manually compile shaders for a particular datablock? In my case at least, this would allow me to effectively work around the issue - as I'd be able to, when first creating the datablock, see if it compiles, and then if it errors out, unload it and return some preconfigured default datablock instead.

rpgplayerrobin
Bugbear
Posts: 805
Joined: Wed Mar 18, 2009 3:03 am
x 467

Re: More graceful invalid shader handling + fallback shaders

Post by rpgplayerrobin »

I solved this now actually.
It is rather simple.

In my game, I press F5 to reload all files that has changed (and I see that they changed from their old modified date), including hlsl files (shaders).

When a shader file is detected to have changed, I simply remove it from the microcache and then reload it, which then recompiles the shader with its new code.
However, this is where it also crashes if it failed to compile, as detailed in this post.

But what I did now to fix it is to create a temporary copy of that shader and then try to compile that first, while also catching its exception with the exact error.
If it fails to compile, the real shader is untouched and will still have the latest working version, but if the temporary shader compilation succeeds, I can then recompile the real shader.
This approach successfully solves this whole problem, and you can easily output the error in any way you wish.

Here is the full code:

Code: Select all

uint32 _getHash(std::string source, std::string programName, uint32 seed = 0)
{
	uint32 hash = FastHash(programName.c_str(), programName.length(), seed);
	return FastHash(source.c_str(), source.length(), hash);
}

void GetShaderMicrocodeCache(GpuProgramPtr gpuProgramPtr, std::vector<uint32>& microcodes)
{
	// Get the microcodes from the shader
	if (CGeneric::m_renderSystem == CGeneric::eRenderSystem::Direct3D11)
	{
		uint32 seed = FastHash("D3D11", 5);
		uint32 hash = _getHash(gpuProgramPtr->getSource(), gpuProgramPtr->getName(), seed);
		microcodes.push_back(hash);
	}
	else
	{
		// Note that this most likely does not work for OpenGL. It only works for D3D9.
		uint32 hash = _getHash(gpuProgramPtr->getSource(), gpuProgramPtr->getName());
		microcodes.push_back(hash);
	}
}

void RemoveShaderMicrocodeCache(GpuProgramPtr gpuProgramPtr)
{
	// Check if microcode is being used
	if (GpuProgramManager::getSingleton().getSaveMicrocodesToCache())
	{
		// Get the microcodes from the shader
		std::vector<uint32> tmpMicrocodes;
		GetShaderMicrocodeCache(gpuProgramPtr, tmpMicrocodes);

	// Loop through all microcodes
	for (size_t i = 0; i < tmpMicrocodes.size(); i++)
	{
		// Check if the microcode exists in the cache
		uint32 tmpMicrocode = tmpMicrocodes[i];
		if (GpuProgramManager::getSingleton().isMicrocodeAvailableInCache(tmpMicrocode))
			// Remove the microcode from the cache
			GpuProgramManager::getSingleton().removeMicrocodeFromCache(tmpMicrocode);
	}
	tmpMicrocodes.clear();
}
}

...


// Check if a GPU program exists for the shader
GpuProgramPtr tmpGpuProgramPtr = GpuProgramManager::getSingleton().getByName(fileName);
if (tmpGpuProgramPtr)
{
	// Clone the shader
	HighLevelGpuProgramPtr tmpClonedGpuProgramPtr = HighLevelGpuProgramManager::getSingleton().createProgram(
		tmpGpuProgramPtr->getName() + "_TMP",
		tmpGpuProgramPtr->getGroup(),
		tmpGpuProgramPtr->getLanguage(),
		tmpGpuProgramPtr->getType());

tmpClonedGpuProgramPtr->setSourceFile(tmpGpuProgramPtr->getSourceFile());

ParameterList tmpParameterList = tmpGpuProgramPtr->getParameters();
std::vector<std::string> tmpNames;
std::vector<std::string> tmpValues;
for (size_t i = 0; i < tmpParameterList.size(); i++)
{
	const ParameterDef& tmpDef = tmpParameterList[i];
	tmpNames.push_back(tmpDef.name);
	tmpValues.push_back(tmpGpuProgramPtr->getParameter(tmpDef.name));
}
for (size_t i = 0; i < tmpNames.size(); i++)
	tmpClonedGpuProgramPtr->setParameter(tmpNames[i], tmpValues[i]);

// Remove the microcache, otherwise it will not compile the new file version
RemoveShaderMicrocodeCache(tmpClonedGpuProgramPtr);

bool tmpSucceeded = true;
try
{
	// Compile the temporary shader
	tmpClonedGpuProgramPtr->load();
}
catch (Ogre::Exception &e)
{
	// The compile failed
	tmpSucceeded = false;
	MessageBox(NULL, e.getDescription().c_str(), "Shader could not compile!", MB_OK | MB_ICONERROR | MB_TASKMODAL);
}

// Destroy the temporary shader
String tmpStr = tmpClonedGpuProgramPtr->getName();
HighLevelGpuProgramManager::getSingleton().unload(tmpStr);
HighLevelGpuProgramManager::getSingleton().remove(tmpStr);
tmpClonedGpuProgramPtr.reset();
if (MeshManager::getSingleton().resourceExists(tmpStr))
	MessageBox(NULL, "The removal of the temporary shader resource failed, the resource is still in use somewhere else.", "", MB_OK);




// Check if the temporary shader could be compiled
if (tmpSucceeded)
{
	// Remove the microcode from the cache
	RemoveShaderMicrocodeCache(tmpGpuProgramPtr);

	// Reload the program (which also compiles it from its new source file)
	tmpGpuProgramPtr->reload();

	// Output a message about the file reload
	CString tmpSourceFile = tmpGpuProgramPtr->getSourceFile();
	DebugOutput("Shader reloaded: " + tmpCurrentFile.fileName + " (" + tmpSourceFile + ")");
}
}
paroj
OGRE Team Member
OGRE Team Member
Posts: 2283
Joined: Sun Mar 30, 2014 2:51 pm
x 1246

Re: More graceful invalid shader handling + fallback shaders

Post by paroj »

haloman30 wrote: Mon Sep 08, 2025 2:31 am

Would it be easier/simpler to have some mechanism to manually compile shaders for a particular datablock? In my case at least, this would allow me to effectively work around the issue - as I'd be able to, when first creating the datablock, see if it compiles, and then if it errors out, unload it and return some preconfigured default datablock instead.

in Ogre1, this again can be handled by reloading the material. If your shader fails to load, you can use this callback to generate a different shader:
https://ogrecave.github.io/ogre/api/lat ... ae43a86f02

actually, the whole RTSS is built around this mechanism.

rpgplayerrobin
Bugbear
Posts: 805
Joined: Wed Mar 18, 2009 3:03 am
x 467

Re: More graceful invalid shader handling + fallback shaders

Post by rpgplayerrobin »

I think my approach is better, since then you can handle it in just one code block after his users have changed a shader, instead of having to handle materials and different callbacks.

My approach also gives the user the exact error instantly in that code, compared to having to get it later on through a callback (if it is even possible there).

My approach also keeps the last usable shader for the object, which I think is much better than just showing an invalid material.

It is also strange to handle it per material, as a shader is usually used on many different materials, so just using the shader approach that I use seems to just be easier.