[GSoC 2011] Binary format support for scripts

Threads related to Google Summer of Code
Post Reply
Dionis_mgn
Gnoblar
Posts: 7
Joined: Sun Jan 16, 2011 10:26 am
Location: Chelyabinsk, Russia

[GSoC 2011] Binary format support for scripts

Post by Dionis_mgn »

Personal detailes:
Name: Denis Glebov
e-mail/jabber: dionis.mgn@gmail.com
Forum nick: dionis_mgn

Project proposal:
Abstract:
OGRE has advanced resourse system. OGRE can load resourses from script files, for comfortable work with it. But, this function is not core functionality of a rendering system. Therefore, it should be took out into plugin and improve it’s functionality by realising saving of text scripts in binary format for speeding up follow loads.
Script serialization has already been implemented in this plugin: http://www.ogre3d.org/forums/viewtopic.php?f=11&t=62487 . But loading of a scripts is still in core.

Successfull solution should meet following conditions:
  • Serializing of compiled scripts into binary files.
  • Do not change interfaces, if it is possible.
  • Easy to use: It should be as easy as just include it in the plugins.cfg to get it worked.
  • Easy future maintaining
  • Do not require original textual scripts
Assumed solution:
There are two approaches:
  • To save real-time representation of resource.
  • To save textual representation of recource.
The second way is preferable because future maintaining will be very easy. To add a new type of resource need only write the appropriate translator.
The problem is reduced to serialize parsed tokens. So, it is necessary to have two types of parsers: binary (ScriptSerializer) and textual(ScriptParser).

Serialization
Currently, there are many serialization libraries. But most of them too complex. Because task serialization token is fairly trivial, serialization is implemented using OGRE::Serializer. This class provides the necessary functionality. Thus, will not require any dependencies.

Other
Loading of the scripts must be in plugin. So, ResourceGroupManager must delegate initializeResourceGroup and initializeAllResourceGroups methods to some other class (a instance of which is located in a plugin). It will "RecourceGroupInitializer" interface. Script Loading Plugin will add a instance of this interface to ResourceGroupManager at startup. It also makes it easier to create custom Initializer (for example, loading of resources via network).
Some sequence diagram:
Image
Legend:
White - user code
Yellow - OGRE classes
Orange - OGRE classes with a modified interface
Green - TODO classes

Schedule:
Before May 16
  • Study documentation
  • Designing solution
Week 1-2 (May 16 - May 29) 2-4 hours per day
  • Isolation of script loader to the plugin
  • Writing overlay translator
Week 3-8 (May 30 - June 30) 0-2 hours per day
Testing/bug fixing. ( I have my exams at this time. So, developing won't be very active)

Week 9-10 (Jule 1 - Jule 17) 3-4 hours per day [mid. term evaluation]
  • Refactoring of CoreDumped's plugin
Week 10-11 (Jule 18 - 31) 3-4 hours per day
  • Integrating binary serializer with textual one
  • Testing
Week 12-13 (August 1 - August 16) 3-4 hours per day
  • Editing of the documentation if necessary
  • Editing some wiki articles
  • Testing
  • Bug fixing
Why I`m The Person For This Project and Why Is OGRE:
I often use OGRE in my activity. I’m interesting in game developing and other multimedia applications. Unfortunatly I don’t have enough knowledge and experience in 3D/rendering , but I want to help to the OGRE, because it shows me all advantages of object-oriented programming and has had a big part in my professional evolution. Also, progress of OGRE helps to my future projects =). That’s why I want to solve this task: it’s not difficult, doesn’t require knowledge of 3D-rendering, but at the same time, successfull solution of this task can make OGRE better.
Last edited by Dionis_mgn on Fri Apr 15, 2011 3:29 pm, edited 5 times in total.
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [GSoC 2011] Binary format support for scripts

Post by Noman »

This is definitely something that we'd like to see in Ogre.
Your application (submission window opens soon) will need to go into more details on how you plan on doing this.

Some things worth discussing :
1) Is the serialization going to be equivalent to the textual representation of objects or to the runtime representation? For example, in materials, inheritance is a script-only feature and is not part of the Ogre::Material object. Will that be preserved?
2) What framework are you going to use to serialize the objects? Write your own? Google protocol buffers?
3) How easy will it be to maintain this feature in the future (for example, materials getting more properties, or a new type of scriptable object entering ogre)?

Try to answer these questions (mostly to yourself, then to us), and start planning out the stages in the project.

Good luck!
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: [GSoC 2011] Binary format support for scripts

Post by CABAListic »

Isn't (most of) this already solved by CoreDumped's plugin?
http://www.ogre3d.org/forums/viewtopic.php?f=11&t=62487

I feel it would be more sensible to polish that plugin and include it into the core than to reinvent the wheel.
Dionis_mgn
Gnoblar
Posts: 7
Joined: Sun Jan 16, 2011 10:26 am
Location: Chelyabinsk, Russia

Re: [GSoC 2011] Binary format support for scripts

Post by Dionis_mgn »

CABAListic wrote:Isn't (most of) this already solved by CoreDumped's plugin?
Hmm. Yes, it is. :? My fault, it was necessary to look this in the search.
CABAListic wrote:I feel it would be more sensible to polish that plugin and include it into the core than to reinvent the wheel.
But... What's missing from this plugin?
Noman wrote:1) Is the serialization going to be equivalent to the textual representation of objects or to the runtime representation? For example, in materials, inheritance is a script-only feature and is not part of the Ogre::Material object. Will that be preserved?
The first variant is easy to implement and requires no further maintenance, but has a slightly lower performance than the latter. Thus, I tend to text equivalent. But as noticed CABAListic, this approach has already been implemented.
Noman wrote:2) What framework are you going to use to serialize the objects? Write your own? Google protocol buffers?
Write my own. I do not want to add dependencies for such a trifle.
Noman wrote:3) How easy will it be to maintain this feature in the future (for example, materials getting more properties, or a new type of scriptable object entering ogre)?
1. Serialized version of the script will be similar to runtime representation. Then need to modify the implementation of the corresponding serializer.
2. Serialized version of the script will be similar to textual representation. Maintain is not needed.

Likewise, I can rewrite the entire loading scripts using Google Protocol Buffers. This will entail change in the syntax of scripts and additional dependencies. But then the problem of the serialization will be resolved through this library. But is this reasonable?

I will send application as soon as possible. Please, let me know if you decide not to remove the task from the list of ideas for GSoC. :)
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Re: [GSoC 2011] Binary format support for scripts

Post by Praetor »

Ah the scripts... you know re-implementing the scripting systems was my SoC project all those years ago. Now, I'll admit the entire concept rubs me the wrong way. I do not like that Ogre has its very own declarative scripting language. I honestly feel like it causes unnecessary bloat in the code-base.

For our engine we used Lua to script everything, so once the material system's C++ interface was nicely wrapped, we could just use Lua to easily and very nicely declare materials without needing to touch Ogre's scripts. Rinse and repeat for any scripting language you want. Don't like that? Materials can be simply wrapped in callable functions and passed around as .h files and easily included into any Ogre projects.

For me, the amount of maintenance and bloat caused by the scripts outweighs the benefits of having an "official" representation of resources. I am totally willing to be argued against on this one, but I've been thinking about ways that Ogre needs to be streamlined for a while now. And it does need streamlining badly. Its size is large and getting out of control. It's especially noticeable on mobile platforms. They will get more memory in time but I think it would be to our credit if we allowed mobile developers to use their memory footprints more carefully. If not completely cut, systems like the scripts should at least become optional and easily removed from the compiled engine.
Game Development, Engine Development, Porting
http://www.darkwindmedia.com
User avatar
Chris Jones
Lich
Posts: 1742
Joined: Tue Apr 05, 2005 1:11 pm
Location: Gosport, South England
x 1

Re: [GSoC 2011] Binary format support for scripts

Post by Chris Jones »

I do not like that Ogre has its very own declarative scripting language.
Can't that be pulled out into its own Ogre Component? Gradually shrink Ogre's size by pulling anything that can be an optional extra into its own component?
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [GSoC 2011] Binary format support for scripts

Post by Noman »

My two cents :

1) The fact that an implementation exists doesn't mean that you have to implement a different one. If you didn't find out about the plugin, it means that not everyone will find it.

2) I agree that if the scripting system can be turned into a component while maintaining backwards-compatibility (recompilation is fine, but code shouldn't be modified) it would be a good change for Ogre.

So, with these two in mind, I think the correct way to approach this project is to integrate the mentioned binary serialization with the textual one, and move them both to a component. This also means that the plugin needs to be standardized (Ogre conventions, documentation etc. I haven't had a look at its code yet) and some refactoring might be necessary.

This modified project might be a bit more modest in terms of scope, but will still help Ogre a lot, so is still welcome.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: [GSoC 2011] Binary format support for scripts

Post by CABAListic »

I wouldn't completely rule out interface breaking changes, though. We are targeting a new major release, and major releases have always been allowed to change interfaces. If they can be avoided, fine, but if not (easily), then do it right.
User avatar
Noman
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 714
Joined: Mon Jan 31, 2005 7:21 pm
Location: Israel
x 2
Contact:

Re: [GSoC 2011] Binary format support for scripts

Post by Noman »

Hi,

I saw that you uploaded your proposal to the google melange site. Can you please edit your first post and put it there as well, so people can easily comment in this forum?

Thanks!
Dionis_mgn
Gnoblar
Posts: 7
Joined: Sun Jan 16, 2011 10:26 am
Location: Chelyabinsk, Russia

Re: [GSoC 2011] Binary format support for scripts

Post by Dionis_mgn »

Proposal was updated.
User avatar
conkienlua
Deactivated User
Posts: 5
Joined: Tue Sep 20, 2011 6:57 pm

Re: [GSoC 2011] Binary format support for scripts

Post by conkienlua »

On the windows platform, AngelCode can generate fnt file in text, xml or binary. Binary is the most smallest. With all the characters in msyh, the text fnt file is 3.20M while the binary fnt file is 572k. It save much space. = = Yet, I found that the Texture is still too big. I can't find any way to make it smaller if I want to support all the characters. SimSun with all the characters in 24px, the png will consume 10M. After the compression of Unity, it becomes 16M, even more bigger.

Or any way to migrate the glyth generating into EZ GUI? The build-in GUI from Unity 3.0 is able to support dynamic charsets, which is generated on the fly by runtime. If EZ GUI can do such things, the package will even smaller.
Post Reply