UPDATED: Plugin to save/load binary script format

A place to show off your latest screenshots and for people to comment on them. Only start a new thread here if you have some nice images to show off!
User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

UPDATED: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 1:30 am

Hi, I created a plugin that automatically saves the compiled script from memory (the Abstract Syntax Tree) to disk. Next time, when the script is about to be parsed and if its corresponding binary version is available, the compilation is skipped and the AST is loaded directly from the binary file and sent to the translator, which would speed up the loading process. This was pretty straight forward with Ogre's new script compiler.

The following scripts can be saved in binary format:
*.material
*.program
*.compositor
*.particle

There is no need for any code change. Just include this plugin to your plugins.cfg and it will automatically save the compiled script to disk when you run it the first time. Then on subsequent runs, the compiled version would be used from the cache, if available. I noticed that the the loading time of the sample browser was reduced from 7 seconds to 4.5 in my Core i7.

The text based scripts can also be replaced by the compiled version by removing their entries and adding the cache folder in resources.cfg. This could be useful when you dont want end-users to modify your scripts

Image

Code released under MIT license
http://code.google.com/p/ogre-script-serializer/

The default location of the cache directory is ".scriptCache" in the working directory. This directory would be created if its unavailable.
The code was tested to an extent. The AST generated from the script compiler matches with one save and loaded from disk, for all the scripts in the sample media folder. All samples seem to work fine.

Pending Stuff:
1. [s]The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?[/s] - DONE
2. [s]Save/Load compiled shaders to disk[/s] - Already implemented in 1.8 by Assaf Raman. Using this lib in the plugin


EDIT 1: Check this post for details on the new update
Last edited by CoreDumped on Fri Jan 14, 2011 5:26 pm, edited 4 times in total.
0 x

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4143
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 268
Contact:

Re: Plugin to save/load binary script format

Post by dark_sylinc » Thu Jan 13, 2011 2:04 am

Wow! this looks surprising! Congrats!
CoreDumped wrote: Pending Stuff:
1. The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?
MD5 could counter the benefit because you would need to read all script files, and HDD can be a bottleneck (I wonder if it actually is). So a time-stamp could probably be a lot faster, but this might be countered if the HDD must seek too much.
MD5 may benefit during development, since often a script may be reedited multiple times, only to be reverted to the way it was before.
Timestamps on the other hand, would benefit more the final release if the executable allows modding or something like that. And those who don't, don't need source file changes checks at all; just use the binary version.

So, my suggestion is to check timestamps, and if it's newer, compute MD5; with possibility of disabling one or both techniques.
Of course, what you really are willing to implement is up to you ;)

I would go for CRC32 though, it's cheaper to implement, 4 times smaller, faster to compute, and it's OK as long as you don't use it to as a unique hash to compare against multiple files (chance of having a collision between two files is of 1 / 2^32, but if you start comparing against multiple files to see if they're "identical", the birthday paradox makes your chance alarmingly grow)

Keep up the good work!
0 x

User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
Contact:

Re: Plugin to save/load binary script format

Post by Jabberwocky » Thu Jan 13, 2011 3:36 am

An excellent and welcome feature!
0 x
Image

User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

Re: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 7:50 am

Thanks guys.

dark_sylinc, you are right. I'll go with the timestamp approach. However, I'm not sure how I can fetch the file stat of an ogre resource, since it might come from a filesystem, zip or even a remote web server. Any ideas? If it's not feasible, then we can go with CRC32.

Also, to check if the compiled version is valid, I'll also have to check if any of the base scripts (all the way to the top) were modified since the AST generated by the compiler is self-contained which contains the compiled code of the base scripts as well

EDIT: I'm not sure about the last point that the AST contain code of the base script
0 x

User avatar
so0os
Bugbear
Posts: 833
Joined: Thu Apr 15, 2010 7:42 am
Location: Poznan, Poland

Re: Plugin to save/load binary script format

Post by so0os » Thu Jan 13, 2011 8:59 am

You're my saviour, many thanks!
0 x
Sos Sosowski :)
http://www.sos.gd

User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
Contact:

Re: Plugin to save/load binary script format

Post by jacmoe » Thu Jan 13, 2011 10:44 am

This is really great! :D
Resource parsing/loading is really a time sink.
0 x
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.

User avatar
syedhs
Silver Sponsor
Silver Sponsor
Posts: 2702
Joined: Mon Aug 29, 2005 3:24 pm
Location: Kuala Lumpur, Malaysia
x 3

Re: Plugin to save/load binary script format

Post by syedhs » Thu Jan 13, 2011 11:49 am

I really think this should be part of Ogre core - if I am not mistaken there are some discussions toward doing this but nothing comes out from it..
0 x
A willow deeply scarred, somebody's broken heart
And a washed-out dream
They follow the pattern of the wind, ya' see
Cause they got no place to be
That's why I'm starting with me

User avatar
so0os
Bugbear
Posts: 833
Joined: Thu Apr 15, 2010 7:42 am
Location: Poznan, Poland

Re: Plugin to save/load binary script format

Post by so0os » Thu Jan 13, 2011 12:13 pm

syedhs wrote:I really think this should be part of Ogre core - if I am not mistaken there are some discussions toward doing this but nothing comes out from it..
I strongly approve! It takes approx 17 seconds on i5 to load all my materials and stuffs (and thus, boot the app) it's a narrow bottleneck, do it.
0 x
Sos Sosowski :)
http://www.sos.gd

User avatar
spookyboo
Silver Sponsor
Silver Sponsor
Posts: 1140
Joined: Tue Jul 06, 2004 5:57 am
x 15
Contact:

Re: Plugin to save/load binary script format

Post by spookyboo » Thu Jan 13, 2011 1:20 pm

I would even go so far to move all Ogre scripting stuff to the plugin. Serializing/deserializing is not core functionality of a rendering system.
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Plugin to save/load binary script format

Post by CABAListic » Thu Jan 13, 2011 1:36 pm

spookyboo wrote:I would even go so far to move all Ogre scripting stuff to the plugin. Serializing/deserializing is not core functionality of a rendering system.
Interesting idea. It has some appeal to me :)
0 x

User avatar
Zonder
Ogre Magi
Posts: 1133
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 22

Re: Plugin to save/load binary script format

Post by Zonder » Thu Jan 13, 2011 2:38 pm

dark_sylinc wrote:Wow! this looks surprising! Congrats!
CoreDumped wrote: Pending Stuff:
1. The compiled version should be discarded whenever the source file is modified. We can save the MD5 hash of the source file in the binary header and compare before reloading later. Or use the timestamp approach?
MD5 could counter the benefit because you would need to read all script files, and HDD can be a bottleneck (I wonder if it actually is). So a time-stamp could probably be a lot faster, but this might be countered if the HDD must seek too much.
MD5 may benefit during development, since often a script may be reedited multiple times, only to be reverted to the way it was before.
Timestamps on the other hand, would benefit more the final release if the executable allows modding or something like that. And those who don't, don't need source file changes checks at all; just use the binary version.

So, my suggestion is to check timestamps, and if it's newer, compute MD5; with possibility of disabling one or both techniques.
Of course, what you really are willing to implement is up to you ;)

I would go for CRC32 though, it's cheaper to implement, 4 times smaller, faster to compute, and it's OK as long as you don't use it to as a unique hash to compare against multiple files (chance of having a collision between two files is of 1 / 2^32, but if you start comparing against multiple files to see if they're "identical", the birthday paradox makes your chance alarmingly grow)

Keep up the good work!
The last modified date should be enough during development
0 x
There are 10 types of people in the world: Those who understand binary, and those who don't...

User avatar
boyamer
Orc
Posts: 459
Joined: Sat Jan 24, 2009 11:16 am
Location: Italy

Re: Plugin to save/load binary script format

Post by boyamer » Thu Jan 13, 2011 3:40 pm

Absolutelly, this should go to Ogre trunk :)
0 x

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4143
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 268
Contact:

Re: Plugin to save/load binary script format

Post by dark_sylinc » Thu Jan 13, 2011 4:26 pm

+1 to put this in the trunk.
CoreDumped wrote:Thanks guys.

dark_sylinc, you are right. I'll go with the timestamp approach. However, I'm not sure how I can fetch the file stat of an ogre resource, since it might come from a filesystem, zip or even a remote web server. Any ideas? If it's not feasible, then we can go with CRC32.
Archive::getModifiedTime?
ResourceGroupManager::resourceModifiedTime?
0 x

User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

Re: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 4:30 pm

dark_sylinc wrote:Archive::getModifiedTime?
ResourceGroupManager::resourceModifiedTime?
Ah great. Thanks, that would do
0 x

User avatar
so0os
Bugbear
Posts: 833
Joined: Thu Apr 15, 2010 7:42 am
Location: Poznan, Poland

Re: Plugin to save/load binary script format

Post by so0os » Thu Jan 13, 2011 4:55 pm

Hmm... i'd hate to complain, but I didn't get any boot speedup (or a minimal one) from using the plugin, Ogre.log says it skips scripts and processes sbins, but it still takes a year. Am I mising something, or is speedup not the intent here?
0 x
Sos Sosowski :)
http://www.sos.gd

User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

Re: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 6:35 pm

so0os wrote:Hmm... i'd hate to complain, but I didn't get any boot speedup (or a minimal one) from using the plugin, Ogre.log says it skips scripts and processes sbins, but it still takes a year. Am I mising something, or is speedup not the intent here?
I tested this in release mode and see that the compiler's script parsing speed is fast (my original stats were on debug mode). However the ratio is still the same and the difference would be noticeable if you have many scripts

With a large material file with 140k lines, 2000 materials with 2 passes each takes 6 seconds to parse normally, while the plugin takes 3 seconds. I think the translators take up that 2-3 seconds. I'll post proper profiling results later

However, we'll probably wont have that many materials in a game :) Script compilation probably wont take that much time to begin with anyway. I think the real bottleneck is the shader code compilation. I am working on passing the compiled shader code to the GpuProgram through a workaround (i dont see any directly interface to load from binary)

Also, I just updated the code with some optimizations. I was seeking past the header on every AST node, which meant thousands of slow seeks. Refactoring the data structures doesn't require this anymore.
0 x

Rambus
Greenskin
Posts: 100
Joined: Tue Aug 01, 2006 6:50 am
Location: Canada
Contact:

Re: Plugin to save/load binary script format

Post by Rambus » Thu Jan 13, 2011 7:15 pm

This is a very useful and important plugin. If you continue to provide it with some love over the next few days (like serializing compiled shaders) then I think it will get wide use.

Good job, and thank you!
0 x

User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

Re: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 7:16 pm

Rambus wrote:This is a very useful and important plugin. If you continue to provide it with some love over the next few days (like serializing compiled shaders) then I think it will get wide use.

Good job, and thank you!

Thanks :)


Added support for re-compiling the script if it was modified since last compile

Code updated in repository

EDIT: Also added version number in the script header to track data structure changes. So if you are trying to load an older binary version, it will throw an exception. Be sure to clear out your scriptCache folder after the update (rev 6)
0 x

User avatar
spookyboo
Silver Sponsor
Silver Sponsor
Posts: 1140
Joined: Tue Jul 06, 2004 5:57 am
x 15
Contact:

Re: Plugin to save/load binary script format

Post by spookyboo » Thu Jan 13, 2011 8:21 pm

I saw that it supports the basic Ogre script extensions. Can I also add other types? (i.e. *pu or *osm files)
0 x

reptor
Ogre Magi
Posts: 1120
Joined: Wed Nov 15, 2006 7:41 pm
Location: Finland

Re: Plugin to save/load binary script format

Post by reptor » Thu Jan 13, 2011 8:30 pm

Nice! This is clearly a good idea. Thank you for your work.


Did you notice Mr. Assaf Raman's work, a shader cache?
0 x

User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

Re: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 8:33 pm

spookyboo wrote:I saw that it supports the basic Ogre script extensions. Can I also add other types? (i.e. *pu or *osm files)
I was thinking of trying the *.pu scripts out yesterday. I have a license of your library :)

As long as you use the ogre script compiler, it should be transparent and should work

However, for automatic detection of the binary scripts from resources.cfg file, i.e. if you deploy only the binary script, i hardcoded the search extensions to be either one of the four (program, material, particle, compositor). I'll move this to a config file.
Last edited by CoreDumped on Thu Jan 13, 2011 10:59 pm, edited 1 time in total.
0 x

User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

Re: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 9:02 pm

reptor wrote:Nice! This is clearly a good idea. Thank you for your work.


Did you notice Mr. Assaf Raman's work, a shader cache?
Yep, I briefly worked with RTShaderLib while trying out the iPhone ES 2.0 renderer. Its a great lib. Not sure if I will find something related there since the shader source code is cached there and we want to cache the binary format. I'll check it out though
0 x

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4143
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 268
Contact:

Re: Plugin to save/load binary script format

Post by dark_sylinc » Thu Jan 13, 2011 11:11 pm

I think most developers here will agree that improving debug performance time is very important, regardless of how much Release mode actually improves.
It's awful when you're debugging something that took you to restart your application more than 3 or 4 times.

As for the shader binary, there's no abstract interface because all formats are too different.
In GLSL for example, you don't even get the binary or compiled representation at all, it's hidden from you inside OpenGL.
I believe HLSL starting from shader model 4.0 has the same issue.

You would need to cast the shader programs to the adequate derived class (HlslProgram, CgProgram, GLSLProgram, etc) and get direct access to their pointers.
0 x

User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 1
Contact:

Re: Plugin to save/load binary script format

Post by Wolfmanfx » Thu Jan 13, 2011 11:16 pm

If you want to save shaders to the cache:
* Before the resource load call GpuProgramManager::getSingleton().setSaveMicrocodesToCache(true);
* When you want to save the cache to a file call - GpuProgramManager::getSingleton().saveMicrocodeCache(...);

When you want to use the cache:
* Before the resource load call - GpuProgramManager::getSingleton().loadMicrocodeCache(...);
http://www.ogre3d.org/forums/viewtopic.php?f=4&t=61004

I think Assaf did a shader cache which saves the bin representation of the shader.
0 x

User avatar
CoreDumped
Gremlin
Posts: 177
Joined: Sun Aug 05, 2007 3:55 pm

Re: Plugin to save/load binary script format

Post by CoreDumped » Thu Jan 13, 2011 11:26 pm

Wolfmanfx wrote:I think Assaf did a shader cache which saves the bin representation of the shader.
Oh great :). I didn't know about this. So this is what reptor was talking about
0 x

Post Reply