[GSoC 2013 - accepted] Resource System Redesign

Threads related to Google Summer of Code
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Mon Jul 01, 2013 11:30 pm

Thanks. I'll try and get that change pushed soon.

---

I'm currently working on migrating Texture over to the pluggable resource system. Some things I've observed while working on it:
  • The depth member is re-purposed to store the subtexture count for texture arrays, which greatly increases the complexity of some logic. I'm going to separate this out into a separate member
  • The Texture object deals with the source, desired and actual formats and size. Because it is no longer relevant to OgreMain, I'm going to remove the source portion of the equation from Texture. In addition, my intention is to remove the desired and actual distinction: the uploader will set the properties they want, then call "_determineInternalFormat()". The render system will attempt to match the desired format as best possible, but will replace the properties of the texture object with the selected properties
In addition, one of the comments that has been made is that the existing "allocate - map - upload" model complicates the DirectX11 implementation significantly. While I'm overhauling this area, it makes sense to change that.
My current plan is
  • Implement an _uploadBufffers method which provides a "create-and-upload-in-one" solution similar to that used by D3D11.
  • Mandate the use of _uploadBuffers for TU_STATIC[_WRITE_ONLY?] textures
The _uploadImages function will be modified to use _uploadBuffers; it will be the convenience method I expect most resource system implementations to use. I'm also adding an _allocateBuffers method which both allows resource systems to allow the texture manager to allocate the buffer, and additionally allows the render system to optimize memory use. For example, an OpenGL implementation supporting GL_APPLE_client_storage might use this to retain its' own copy in system memory rather than letting the OpenGL implementation do so; and the Direct3D9 render system might return the internal buffers it uses to handle device losses. In cases where _allocateBuffers is used, the client must pass exactly those buffers to _uploadBuffers.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Mon Jul 08, 2013 10:34 pm

Working through things, but progressing slowly. My initial phase has been conversion of OgreMain plus one render system over to the new design. I picked D3D9 as the first render system, because I'm familiar with OpenGL but not with D3D, and wanted to avoid accidentally introducing "OpenGL-isms" to the design, and wanted to work on a "stable" render system (to minimize the number of merge conflicts generated), which ruled out D3D11.

I'm coming to regret that decision, because internally the D3D9 render system is perhaps the most convoluted; the D3D9 render system internally maintains multiple devices for reasons I haven't yet been able to establish. Because of this, it interacts with the resource system in ways which have a large impedance mismatch from the ways in which the modularized design will do so.

This odd feature aside (none of the D3D11, GL, GL3 or GLES render systems maintains multiple contexts), the changes I've been making so far seem to bring things broadly in line with the D3D (and particularly D3D11) model. Because of this, I'm strongly contemplating moving over to doing the initial conversion on the GL render system.
0 x

User avatar
holocronweaver
Google Summer of Code Student
Google Summer of Code Student
Posts: 273
Joined: Mon Oct 29, 2012 8:52 pm
Location: Princeton, NJ

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by holocronweaver » Tue Jul 09, 2013 2:13 am

Owen wrote:Because of this, I'm strongly contemplating moving over to doing the initial conversion on the GL render system.
Hello there. I did not realize you were modifying the render systems for your GSoC. Since some of my work on GL3+ involves patching the GL render system, I am curious what sort of changes you have in mind.
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Tue Jul 09, 2013 5:55 am

I'm with holocronweaver there. Besides the multiple contexts in DX9 causing issues. What issues are you running into? What changes are you planning(go into detail)?
0 x

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Klaim » Mon Jul 15, 2013 9:55 am

Owen, Just in case you are not aware yet, I want to point you to this discussion: http://www.ogre3d.org/forums/viewtopic. ... 65#p492864
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Wed Jul 17, 2013 6:18 am

Owen, How are things going right now? What are you currently working on? Don't forget to give everyone status updates here in the forum.
0 x

User avatar
Zonder
Ogre Magi
Posts: 1129
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 20

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Zonder » Wed Jul 17, 2013 7:44 am

It's nice to have a daily report on progesss for all us bored people at work :)
0 x
There are 10 types of people in the world: Those who understand binary, and those who don't...

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Klaim » Mon Jul 22, 2013 10:51 am

Does Owen is having problems? The last push in the public repo is from June and he is supposed to pause next week. It's a bit worrying. masterfalcon did you get some news?
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Tue Jul 23, 2013 4:38 am

Sorry for the lack of updates. My laptop charger failed to do what laptop chargers are supposed to do; namely, charge my laptop. As you might guess this was quite an impediment to programming! Also, I thought I'd posted an update a while back. I guess I drafted it, left it in a tab overnight to proofread it and promptly lost it...

Currently working my way through GPUProgram and Compositors. A lot of the core changes are done; ResourceManager is much slimmed and might get renamed to ResourceFactory to reflect its' new nature.; ResourceGroupManager is gone. Textures are mostly converted but I want to do another pass over them because

Mostly I've hacked out the old code and am going around cleaning up the dust. Mostly it involves looking at the error list, fixing a batch of errors, compiling, sword fighting on wheelie chairs, then moving onto the next set of errors.

Now that I'm more confident that the internal Resource is "locked down", I'll start separating out the changes and pushing them to Hg.

A lot of the stage I've reached now is purely deleting code (which will come back as a plugin or library later).

OgreMain and OpenGL should hopefully be building again within a week. The next step after that will be a minimal app which displays a textured triangle or similar, entirely "by hand" implementing its' own ResourceLoaders. That shouldn't take long. The variable after that is bug fixing; some of the interactions are complex; I don't pretend I'll have got them correct first time.

After that it's all (re)construction work. Being as this involves a whole lot less of tracing twisting code paths across multiple classes, this should proceed significantly faster.

While the GSoC schedule nominally has a 1 week gap, in practice I'll be continuing onwards anyway.
0 x

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Klaim » Tue Jul 23, 2013 10:05 am

Nice to hear from you :D
Textures are mostly converted but I want to do another pass over them because
Is this sentence finished?
0 x

User avatar
holocronweaver
Google Summer of Code Student
Google Summer of Code Student
Posts: 273
Joined: Mon Oct 29, 2012 8:52 pm
Location: Princeton, NJ

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by holocronweaver » Tue Jul 23, 2013 5:30 pm

Owen wrote: OgreMain and OpenGL should hopefully be building again within a week.
Still curious about what changes you are making to the OpenGL render system.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Tue Jul 23, 2013 9:52 pm

Klaim wrote:Nice to hear from you :D
Textures are mostly converted but I want to do another pass over them because
Is this sentence finished?
Oops. This is why I usually leave these posts to sit overnight.

...because theres one aspect of the model I'm not sure is is sufficiently general.
holocronweaver wrote:
Owen wrote: OgreMain and OpenGL should hopefully be building again within a week.
Still curious about what changes you are making to the OpenGL render system.
Resource subclasses (Texture, GPUProgram) are getting adjusted to fit the new resource model. That is all.
0 x

User avatar
holocronweaver
Google Summer of Code Student
Google Summer of Code Student
Posts: 273
Joined: Mon Oct 29, 2012 8:52 pm
Location: Princeton, NJ

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by holocronweaver » Tue Jul 23, 2013 9:59 pm

Owen wrote:Resource subclasses (Texture, GPUProgram) are getting adjusted to fit the new resource model. That is all.
Sounds good - I do not believe I have touched those classes in the GL RS, so we shouldn't have any conflicts. Will let you know if this changes.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Wed Jul 24, 2013 12:47 am

I've written up a "birds-eye tour" of the changes, aimed as an introduction to them.
0 x

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Klaim » Wed Jul 24, 2013 1:13 am

Very nice read! :D
0 x

User avatar
holocronweaver
Google Summer of Code Student
Google Summer of Code Student
Posts: 273
Joined: Mon Oct 29, 2012 8:52 pm
Location: Princeton, NJ

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by holocronweaver » Wed Jul 24, 2013 2:09 am

Thanks for this - I understand the new resource system much better now.

By the by, apparently the SkyDrive service aliases the text horribly when viewed on the web. I never have aliasing issues, so I can only assume this is intentional to force a MS Office purchase. I humbly ask that you use another service for us non-MS users (even Dropbox has a better viewer).
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Wed Jul 24, 2013 4:05 am

holocronweaver wrote:By the by, apparently the SkyDrive service aliases the text horribly when viewed on the web. I never have aliasing issues, so I can only assume this is intentional to force a MS Office purchase. I humbly ask that you use another service for us non-MS users (even Dropbox has a better viewer).
I checked the web version on both Windows, OS X and Android and there was no text aliasing (admittedly the diagram fell apart on the mobile version, but that's unrelated). I suspect that for whatever reason the document is triggering a bad font on your system.

In any case, if you go to print, it'll render a PDF of it as a workaround.

The Office Web Apps are a product actually sell (as part of Office 365 and Sharepoint); it's actually not in their interests to criple it
0 x

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Klaim » Wed Jul 24, 2013 8:01 am

I use google drive personally.
0 x

User avatar
Zonder
Ogre Magi
Posts: 1129
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 20

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Zonder » Wed Jul 24, 2013 11:13 am

The document worked fine for me as well. Like mentioned I bet it's using a dodgy font. Maybe format the document in ariel? That is pretty safe across different platforms.
0 x
There are 10 types of people in the world: Those who understand binary, and those who don't...

drwbns
Orc Shaman
Posts: 777
Joined: Mon Jan 18, 2010 6:06 pm
Location: Costa Mesa, California

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by drwbns » Wed Jul 24, 2013 12:25 pm

Works fine for me too, different font than Ogre forums but reads ok after some zooming :)
0 x

User avatar
quiasmo
Kobold
Posts: 28
Joined: Thu Jun 03, 2010 6:59 pm

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by quiasmo » Wed Jul 24, 2013 5:28 pm

Hey there.

A useful feature for when writing editors is to know where the resource in use is coming from.
This could be a file, zip archive, or manual.
If it comes form a file, knowing which file this comes from is quite a complicated task.
However a small change in the resource group manager helps out and doesn't seemingly sacrifice any design principles.

Change all methods named resourceExists

from

Code: Select all

bool Ogre::ResourceGroupManager::resourceExists(...);
to

Code: Select all

Archive * Ogre::ResourceGroupManager::resourceExists(...);
And return the found archive accordingly or null otherwise.

Hope this bit of useful info can shape a similar feature in the new system.
Thanks.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Thu Jul 25, 2013 3:48 am

Things have been going very well the last couple of days. The remaining changes are starting to coalesce around Ogre::Root and Ogre::SceneManager, with only minimal functionality stubbed out (e.g. texture aliasing). The end is very much in sight (for this phase)

Other changes of note:
  • I've added logic to detect nullptr support. On compilers with it, Ogre::nullptr_t is aliased to std::nullptr_t, and OGRE_NULL is defined to nullptr. On compilers without, a C++03 implementation of nullptr_t is implemented as Ogre::nullptr_t, and OGRE_NULL defined to (::Ogre::nullptr_t()). Implicit convertibility works for this as it does for the C++11 builtin.
  • Ogre::IntrusivePtr added, which couples more closely with the resource system logic. That said, down the line I can see this being replaced by an even more tightly coupled option in a later stage; consider a smart pointer which can tell the resource system whether it's holding onto the resource as a handle or for rendering purposes
  • Both IntrusivePtr and SharedPtr now implement the safe bool idiom, meaning you can do if(ptr) like you can on normal pointers
  • Both IntrusivePtr and SharedPtr's default constructors have been redefined as XxxPtr(nullptr_t _ = OGRE_NULL), with the result that you can pass 0/NULL/OGRE_NULL/nullptr to a function which takes one of them like you can with normal pointers
Regarding the above changes: is there interest in taking them up (IntrusivePtr aside) into the current 2.0 branch?

Regarding the meat of my work: I'm very confident all the interfaces are locked down now, so I'll start pushing the work. With any luck, I'll be done with OgreMain by tomorrow (25th); without, likely the day after (26th).

Once that has been done, as stated the next priority will be getting a test app up in order to resolve the bugs that will inevitably have crept in. That test app will start with a prefab plane with the default material, and proceed from there. I'll get to the stage where meshes, textures and materials work in that test app; after that, I'll work on implementing the new-old default resource system and bringing the samples back up.

I should be able to port across the GL/GL3/GLES render systems myself, and the DX11 render system doesn't look too bad. The DX9 render system I might need assistance from someone more familiar (I'm both unfamiliar with DX9, and its' somewhat cruftier than DX11. Mostly, this involves dealing with devices with differing capabilities).
0 x

User avatar
holocronweaver
Google Summer of Code Student
Google Summer of Code Student
Posts: 273
Joined: Mon Oct 29, 2012 8:52 pm
Location: Princeton, NJ

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by holocronweaver » Thu Jul 25, 2013 4:33 am

Go, Owen, go! :D
Owen wrote:I should be able to port across the GL/GL3/GLES render systems myself
I was wrong before about not touching Texture and GpuProgram classes in GL3 - I have/will be modifying them fairly extensively, so I ask that you leave GL3+ alone until after the GSoC merge. If you really cannot help yourself, and I can completely understand if that is the case :) , you could merge my branch into yours and then make the changes from there. Let me know if you want to take that route and I will try to get things in order for a merge.
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by masterfalcon » Thu Jul 25, 2013 6:20 am

Sounds great Owen! I don't think we'll merge IntrusivePtr into 2.0 yet until after GSoC. Also don't forget that we have a build farm now(not yet announced, I'll get to that soon). So you can see if it builds on other systems.

For now I would leave GL3+ alone, it closely follows GL and GLES 2 so any changes to them can be easily be applied to GL3+ later. It also may cause merge conflicts. The same thing may apply to DX11, I'd talk to robert_sasu to see if your changes and his may conflict in any way.
0 x

Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am

Re: [GSoC 2013 - accepted] Resource System Redesign

Post by Owen » Thu Jul 25, 2013 3:59 pm

nullptr_t and SharedPtr pull request. I think I got all the conditional guards correct. I suppose the build farm will see...

The resource system split is moving along. Getting the commit stack together shouldn't take too long.
0 x

Post Reply