Added - a cache of the binary representation of shaders

[sparkprime]

Firstly it is more high level and less general -- no talk of passes / techniques, just turning on/off various things like normal mapping and stuff I put in the ubershader. Secondly it's specified in lua which parses much faster than material scripts. There are also global options for turning on / off various things and detail levels, e.g. anisotropic filtering is centrally controlled, requiring all the materials to be rebuilt. I also have more than one ogre material for each of my materials, because I fade things out on LOD transition, which needs a slightly different shader and alpha blending in some cases. There are also special shadow casting materials when a diffuse needs to be bound for alpha rejection.

here is our wiki page for it

http://www.gritengine.com/game-engine-w ... p/Material

[reptor]

Reading text files is fine, even AAA games do it and have done for years. However I did develop my own material system since OGRE's was taking minutes when it should have taken seconds (early 2009).

The commercial games I have had a closer look at all have binary file formats for 'scripts'. They do have the engine accept text files too, but the finished product has binary file formats which load faster. They do this for configuration files, material definitions and game-logic scripts.

I have to say that it did cross my mind some months ago why OGRE has its own scripting system when scripting libraries such as Lua could do this job at least as well if not better and would have more support from a much larger group of people. And Lua has a binary file format for faster loading. Me thinks, perhaps there could be an add-on project which could be used as an alternative to the OGRE scripting system with Lua. Just an idea. It wouldn't be to replace what OGRE has but to be an alternative. So if people decide to use Lua scripting in their application they could use it for the OGRE scripts as well.

This should be in a separate topic. Maybe we should move this to another topic. I apologise to Assaf for going off-topic. Please feel free to move my post.

Oh, and again, the work Assaf has done is superb - I am happy that there are people who work so much on OGRE. Every project needs this kind of people to keep them alive.

[Assaf Raman]

@reptor: Thanks for the kind words. I enjoy programing.
Regarding the scripting - well - I suggest moving the discussion to a new dedicated thread.

[so0os]

Hi,
I recently tried to integrate the Microcode Cache thingy into Oge 1.7.2, but I failed.
Can you give me an approximation of thing's you've changed to get this working? It'd save me a lot of time.

[Assaf Raman]

What render system do you use?

[so0os]

Direct3D 9 exclusively.

[Assaf Raman]

Well, what is the issue you are having?

[Assaf Raman]

Look at changesets: 2463 to 2470 and 2517.
The changes are complex in the gl render system - but in dx they are much simpler, look at changes to the dx render system in changeset 2463.

[so0os]

Thanks a bunch! That's exactly what I needed!

[azenren]

Hi,

Can this be used to get around using d3dcompiler_xx.dll? If I only use the binary rep for my shader in my app, and pretty much "hardcode" the inputs to the single shader that I want to use, can I leave out the d3dcompiler? I need to get MS App Cert., so wondering how to work around it in my case (not the general case), where I have a single vertex shader and a single pixel shader that I know the shader inputs and outputs for.

Thanks,
Renee

[scrawl]

I'm having a problem to get this to work with GLSL. CG works fine, I have noticed the compile time for one shader going down from 1.6 ms to about 0.3 ms. However, when I use the GLSL shaders, the shader simply does not display. The material just appears like in fixed-function. As soon as I remove the shader caching code again (or delete the cached file), the shader works again.

Has anyone successfully got this to work with GLSL shaders?

Here is some interesting fact:

After the first run in GLSL mode (which displays the shaders sucessfully), the cache file is about 20 KB
After the second run in GLSL mode (which shows the materials as fixed function), the cache file is now about 40 KB!
After third run, its 60 KB and so on....

The same thing happens in CG mode, yet the problem does not occur.

posting some code, just in case
Ogre version 1.8 release

Code: Select all

		Ogre::GpuProgramManager::getSingletonPtr()->setSaveMicrocodesToCache(true);

   // load
		std::ifstream inp;
		inp.open(file.c_str(), std::ios::in | std::ios::binary);
		Ogre::DataStreamPtr shaderCache(OGRE_NEW Ogre::FileStreamDataStream(file, &inp, false));
		Ogre::GpuProgramManager::getSingleton().loadMicrocodeCache(shaderCache);


///    my program runs here, and shaders are loaded here .........


 // save on exit

		std::fstream output;
		output.open(file.c_str(), std::ios::out | std::ios::binary);
		Ogre::DataStreamPtr shaderCache (OGRE_NEW Ogre::FileStreamDataStream(file, &output, false));
		Ogre::GpuProgramManager::getSingleton().saveMicrocodeCache(shaderCache);

[scrawl]

This feature seems to be largely broken with GLSL.
I've found that for some users, just calling

Code: Select all

      Ogre::GpuProgramManager::getSingletonPtr()->setSaveMicrocodesToCache(true);

will cause a crash in Ogre::GLSLLinkProgram::compileAndLink line 580 on the first render frame.

Edit: It seems that the problem occurs in 32 bit linux builds, but not in 64 bit.

[Assaf Raman]

No clue, I wrote and checked it in windows.

[delphenium]

This is great! It made startup time of the app go from something like 12 seconds to ~1 second.

[lunkhound]

This is a really great feature. I find it really helpful for compiling my shaders with every possible permutation of compile options (which takes a minute or two with HLSL). I don't want my shipped product to be compiling shaders on end-users' machines, so this shader cache feature solves that issue.

One thing I'd like to be able to do though is recompile a single shader at run time. This would be only for development, when iterating on a shader. It looks like you'd need to call "unload()" on the shader, then somehow remove it from the shader cache, then call "load()". Looking through the code I didn't see any obvious way to remove a single item from the shader cache though. Is it possible without changing Ogre source?

[Assaf Raman]

I can add it, but I think a small code change in OGRE will be the only way to remove.

[lunkhound]

I can add it, but I think a small code change in OGRE will be the only way to remove.

I got it working but had to make a small change in Ogre. I added this:

Code: Select all

   void GpuProgramManager::removeMicrocodeFromCache( const String & name )
   {
		String nameWithRenderSystem = addRenderSystemToName(name);
		MicrocodeMap::iterator foundIter = mMicrocodeCache.find(nameWithRenderSystem);
		if ( foundIter != mMicrocodeCache.end() )
        {
            mMicrocodeCache.erase( foundIter );
        }
   }

There's a slight ickiness in that I have to remember to prefix my shader name with "D3D9_HLSL_" before calling but I can live with that, it's only a development feature. I think this is a very useful addition so I'd like to see it added to the 1.9 branch. I can post a patch on the patch tracker if that helps.