[GSoC 2013 - accepted] Resource System Redesign

Threads related to Google Summer of Code
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

[GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Thu Apr 18, 2013 7:44 pm

Edit: Wiki page and Repository created

I'd like to propose a project to replace the Ogre3D resource manager as outlined in my posts in this thread in the Ogre 2.0 forum

A summary of my proposed changes:
  • Make OgreMain independent of the resource system; providing hooks for an external resource system to integrate itself
  • Move the mesh/material/etc loaders to their own (optional) libraries
  • Reimplement the existing resource system in its own optional library
  • Generalize the existing codec system to all resource formats
(All of the above libraries will become optional libraries in the same manner as the RTSS and similar are now)

Proposed timeline
Bonding Period (-17 June)
Establish current interractions between OgreMain and the resource system (especially those not passing through the Resource class and its' descendants). Establish additional requirements for new architecture from other users. Finalize design of the new resource architecture. Consider design changes to existing resource manager as it is migrated to become the "default resource system"

Preliminaries (~1 week 17 June - 24 June)
Remove all convenience methods in OgreMain taking resource names and provide replacements taking Resource pointers where necessary (e.g. createEntity)
Remove all methods in OgreMain taking resource file paths and provide replacements taking DataStreamPtr or other parameter types where necessary (e.g. setWorldGeometry)
Provide "forward compatibility" stubs for accessing resources matching the future "default resource system" API
Update all samples to use the new forward compatibility stubs for all resource accesses

OgreMain refactor & redesign (~5 weeks 25 June - 29 July)
Move most of the existing resource system (ResourceGroup*, Archive*, etc) to the (initially unbuilt) OrgeResourceManager library
Move the existing resource loaders (Mesh loader, Material loader, etc) into the new Ogre*Loader libraries (built)
Move the existing script parser into the new OgreScriptParser library (built)

Refactor and rewrite ResourceManager and Resource inline with the new resource system design. Create the new ResourceLoader interface. Migrate all existing resource types and managers across to the new interface.

Implement a "trivial/example" resource system (one which vends hard-coded resources) to verify changes to OgreMain

Schedule slack (29 July - 02 August)

Midterm Evaluation (02 August)
At this stage changes to OgreMain should be largely complete. Remaining changes will focus on re-implementing the a resource system with functionality at least equivalent to the existing resource system and implementing libraries with additional functionality

Default Resource System (1.5 weeks 03 August - 13 August)
Modify the existing resource system (separated into the OgreResourceSystem library back in Stage 2) to function with the new OgreMain resource hooks. Bring it back up to at least functional parity with the existing resource system.
Ensure all samples build against this new library

Network Resource System (1.5 weeks 14 August - 24 August)
Implement an "on demand" network loading resource system (perhaps based upon the popular, widely available and liberally licensed libcurl) and place it in the optional OgreNetworkResourceSystem library. Implement a sample which uses this library.

Additional Model Format Loader (1 week 25 August - 31 August)
Add an additional model codec, either based upon the AssImp library or similar, or for a relatively simple and common model format (.x perhaps?).

Merge, Bug Fixing and Schedule Slack (2 weeks 1 September - 16 September)
Time set aside for resolving merge conflicts, finding and fixing bugs, and any overruns in the above, and reserved for any additional suggested projects/stretch goals.

Why me?
I've been using Ogre on and off for years (as you might surmise from my join date), and therefore have a not insubstantial level of familiarity with it. Additionally, I have implemented a threaded, asynchronous resource system with similar principles to the ones I propose for a custom engine in the past. I am additionally very used to programming in both C++ (11 and 03) and C, in a style similar to that used by the Ogre codebase. You can find much of my more recent code on Bitbucket.

Academic background: I'm in my penultimate year of studying Physics at the University of Sheffield.
Last edited by Owen on Sat Jun 01, 2013 4:02 pm, edited 1 time in total.
0 x

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4293
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by spacegaier » Mon May 27, 2013 8:13 pm

Congratulations! This proposal has been selected as one of our this year's GSoC projects!

Best of luck and happy coding :)!


Mentor: David Rogers <masterfalcon>
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

PhilipLB
Google Summer of Code Student
Google Summer of Code Student
Posts: 550
Joined: Thu Jun 04, 2009 5:07 pm
Location: Berlin

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by PhilipLB » Mon May 27, 2013 10:31 pm

Yay, congratz. :)
0 x
Google Summer of Code 2012 Student
Topic: "Volume Rendering with LOD aimed at terrain"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: Mattan Furst


Volume GFX, accepting donations.

TheSHEEEP
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 972
Joined: Mon Jun 02, 2008 6:52 pm
Location: Berlin

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by TheSHEEEP » Tue May 28, 2013 7:45 am

Nice! :)

Good luck with the project :)
0 x
My site! - Have a look :)
Also on Twitter - extra fluffy

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Klaim » Tue May 28, 2013 10:32 am

Congrats! I'll follow this.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Owen » Wed May 29, 2013 12:21 am

Thank you everyone (and congrats to everyone else accepted, and consolations to those not). My last exam is on Friday; I'll be able to start the processes after that.
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Wed May 29, 2013 1:29 am

Hey Owen!

The first thing that we should do is add your outline and timeline to the wiki page(http://www.ogre3d.org/tikiwiki/tiki-ind ... eplacement). Also get familiar with the Ogre programming guidelines http://ogre.svn.sourceforge.net/viewvc/ ... dards.html. And finally, create a fork of Ogre on bitbucket and post the link back here in the first post and on the wiki. I think we should be working out of the 2.0 branch too.
0 x

frostbyte
Orc Shaman
Posts: 737
Joined: Fri May 31, 2013 2:28 am
x 14

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by frostbyte » Fri May 31, 2013 2:43 am

don't know if the discussion is over, so here i go...
please please...-add dynamic resource loading on request - waiting for resources to parse and load is a real bummer...
YES i know it can be done some how by tricking ogre to...bla bla bla...
but i think a lot of people would agree that this is a more important feature for end-users then some inside architectural changes...
any way congrats... and have don't forget to have fun...
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Owen » Sat Jun 01, 2013 4:08 pm

masterfalcon wrote:Hey Owen!

The first thing that we should do is add your outline and timeline to the wiki page(http://www.ogre3d.org/tikiwiki/tiki-ind ... eplacement). Also get familiar with the Ogre programming guidelines http://ogre.svn.sourceforge.net/viewvc/ ... dards.html. And finally, create a fork of Ogre on bitbucket and post the link back here in the first post and on the wiki. I think we should be working out of the 2.0 branch too.
Done (And first post updated). I've also filled out and submitted all of Google's forms; I'll be submitting the CLA as soon as I can get it scanned.

A quick point of discussion: When the resource manager is "evicted", what namespace should it go in? "Ogre::DefaultResourceManager" is both obvious and hideously long :-)
0 x

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4293
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by spacegaier » Sat Jun 01, 2013 4:50 pm

Yes, that is the obvious one and I'd personally rather have a long but meaningful name than something short and cryptic and not fitting the style/conventions of the other Ogre classes.

So +1 for Ogre::DefaultResourceManager
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Klaim » Mon Jun 03, 2013 11:32 am

Great!
Move the mesh/material/etc loaders to their own (optional) libraries
Will it be several different libraries or will you end up putting everything currently default in DefaultResourceManager?
If they are different libraries/modules, will you put them in the same namespace?


I think I will be able to not use the current resource system explicitely for some months (I'm using OgreProcedural so far), so I will also be able to early test this new setup.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Owen » Mon Jun 03, 2013 2:17 pm

I'm thinking several different plugins. Resource systems won't have to use them, but the option will always be there.
0 x

Transporter
Minaton
Posts: 933
Joined: Mon Mar 05, 2012 11:37 am
Location: Germany
x 2

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Transporter » Tue Jun 04, 2013 10:25 am

Congratz! :)

I have a few comments on your subscription.

There are a lot of useful things planned. All parts with external libraries like curl or assimp should be optional to keep the amount of required dependencies as low as possible.
Owen wrote:Network Resource System (1.5 weeks 14 August - 24 August)
Implement an "on demand" network loading resource system (perhaps based upon the popular, widely available and liberally licensed libcurl) and place it in the optional OgreNetworkResourceSystem library. Implement a sample which uses this library.
Curl and libcurl are licensed under a MIT/X derivate license, so this is a good choice. But put this part with low priority. I would prefere the next point to load other graphic formats.
Owen wrote:Additional Model Format Loader (1 week 25 August - 31 August)
Add an additional model codec, either based upon the AssImp library or similar, or for a relatively simple and common model format (.x perhaps?).
You can have a look at OgreAssimp for various file formats. I can also help you for adding OpenCascade to support IGES and STEP files.
0 x

User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 1
Contact:

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Wolfmanfx » Tue Jun 04, 2013 10:36 am

Here are some useful blog posts for you

http://molecularmusings.wordpress.com/2 ... pi-design/
http://molecularmusings.wordpress.com/2 ... pi-design/

What would be great if we could chain Archives. For example we have a custom archive (binary blob) which contains a zip archive and that contains again something else so we could reuse the archives.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Owen » Thu Jun 06, 2013 2:11 am

Looking at the existing SceneManager overloads for createEntity:
  • createEntity(name, const MeshPtr& mesh) calls createEntity(name, mesh.getName(), mesh.getGroup())
  • createEntity(name, meshName, meshGroup) calls createMovableObject(entityName, EntityFactory::FACTORY_TYPE_NAME, {"name": meshName, "group": meshGroup})
  • createMovableObject calls through to EntityFactory::createInstance
  • EntityFactory::createInstance looks up the Mesh again by its name and group (...) and returns OGRE_NEW Entity(name, mesh)
Now, obviously, in a world where resource names are a thing Ogre no longer deals with, this solution is unsatisfactory. My proposal with regards to the createEntity(..., MeshPtr) overloads is to just create the Entity directly.

It leaves the question of what to do with EntityFactory though:
  • Discard it?
  • Move it to the DefaultResourceManager plugin?
  • Something else entirely?
It raises the other question also: If we choose to keep the EntityFactory in some form, do we keep the createEntity overrides which take a resource name (and perhaps group)? On the one hand, they would depend upon the presence of a MovableObjectFactory which may not be present, but on the other they would help to maintain backwards compatibility.

If we did keep them, my inclination would be that they would be devirtualised in any case.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Owen » Wed Jun 19, 2013 12:32 am

After a bunch of fighting with tooling (mainly Mercurial extensions), first changes pushed. These are just some preliminaries. I'm currently working through ~150 calls to functions taking resource names, and categorizing them into those internal to the resource system (which get a short stay of execution) and those not (which are being replaced and removed as appropriate). Once that is done, it will be time to start investigating pieces of code taking resource names (most of which should be going now - an awful lot of code in Ogre extracts a resource's name and group, only to turn it back into a resource 2 function calls later)

Ogre::SubMesh and related are the trickiest components in this regard, and are the main things making the code not build.

I added Ogre::Mesh::getDefault() to replace the pattern of various bits of code grabbing BaseWhiteNoLighting as a "fallback" material. In retrospect, I'm not happy with this name, and perhaps getFallback would be better (duh?)
0 x

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4293
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by spacegaier » Wed Jun 19, 2013 6:57 am

I think getDefault() is fine, since that is the method name used in many Ogre source files and fallback and default can often be used synonymously. Depending on what exactly is set to default/fallback values in that method it might also make sense to choose getDefaultSettings(), if only things like the material are set, but no default mesh (in the sense of vertices and stuff).

That is at least my take on that ;) .
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by Owen » Wed Jun 19, 2013 12:20 pm

I meant Ogre::Material::getDefault(). Gyah.
0 x

User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4293
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 2
Contact:

Re: [GSOC 2013 - accepted] Resource System Redesign

Post by spacegaier » Wed Jun 19, 2013 6:47 pm

Still pro getDefault() :)
0 x
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Thu Jun 20, 2013 7:37 am

Same here. At first I wasn't sure but when you clarified that it would be a Material method then it makes more sense and should be clear.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Thu Jun 20, 2013 10:21 pm

I ended up taking a look under the covers of Ogre::SharedPtr. I knew the implementation was suboptimal.

I didn't realise it was quite that bad. For a threaded, 64-bit build a single instance of SharedPtr weighs in at 40 bytes plus two heap allocations per pointee. Yeech.

Ogre::SmartPtr has a heinous list of crimes:
  • It's virtual. Why is it virtual? I have no idea. It has no reason to be virtual. Ogre can be a bit virtual happy at times; this would be the flag bearer
  • Not combining the reference count, mutex and destruction information into one object
  • Using a mutex
Anyhow, I took a slight break from the resource manager work and fixed it, with a patch against the v2-0 branch. This both slims things down to a respectable 2*sizeof(void*), the best any reasonable implementation of shared_ptr can be, and moves them over to atomics.

It also excises the dozen or so subclasses of shared_ptr which existed just to support Visual C++ 6
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Fri Jun 21, 2013 5:59 am

Great work! I'm taking a look at it now and making some minor changes plus a few additional changes that this allows me to make now, like rearranging when threading headers are included.
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Tue Jun 25, 2013 3:22 pm

Owen,

Just wondering, does your branch currently build for you? I get a lot of errors.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Tue Jun 25, 2013 11:06 pm

No. My personal working copy has various resource name related methods (e.g. ResourceManager::getByName, Resource::getName) made private in order to highlight their uses, and my commits are that work being broken apart using hg record. I think I'm nearly done with the bits of OgreMain which can be done now (i.e. those which aren't intimately tied into the resource system). I intend to get the changes to OgreMain done tonight, and have my mess tidied up by mid-tomorrow at the latest, and then begin work on converting the internals to the new resource system.

I intended to convert the samples and provide forward compatibility stubs, but closer inspection of the structure of the current resource system has made me more unsure as to what the structure of the new default resource system will be, therefore I'm holding off on this until it is in place. Once that is done migration of the samples should be relatively simple.

I'm also currently experiencing some graphical issues. I haven't tracked down the cause yet; my theory is that the factory overclock isn't as stable as hoped (the alternative also plausible possibility is an AMD driver bug).
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Sat Jun 29, 2013 8:43 am

I found the source of the error. It's Material::getDefault. Making it return just a MaterialPtr rather than MaterialPtr& fixes it.
0 x

Post Reply