Resource Manager

Discussion area about developing with Ogre-Next (2.1, 2.2 and beyond)


bstone
OGRE Expert User
OGRE Expert User
Posts: 1920
Joined: Sun Feb 19, 2012 9:24 pm
Location: Russia
x 201

Re: Resource Manager

Post by bstone »

Owen wrote:I don't think it does. It passes them to the work queue which loads one resource (synchronously) per thread in the thread pool. This results in the work queue's threads being tied up loading resources from (potentially slow) sources, even when asynchronous I/O could be used to speed up the process greatly.
Ah, I misread your statement. That's correct.
Owen wrote:You seem to be missing here that my intention is to separate the Resource Manager from OgreMain in its entirety. Resource will become an Ogre-focused perspective on the resource; from the point of view of the rendering system, a resource is either loaded (i.e. in a state ready for use) or unloaded (Unusable; perhaps in RAM or on disk, but from a rendering perspective neither is usable). Note that in my first post I explicitly mentioned that in some cases Ogre would itself cause a resource to transition to the unloaded state (when a device is lost, for example)
My intention is to clean up potential issues. Earlier you mentioned that the RESIDENT state of a resource means it's delivered to the GPU. And the only other proposed states were UNLOADED and LOADING. Hence the concern.
Owen wrote:The default resource manager (i.e. the one shipped with Ogre which provides the functionality built in to OgreMain today) will continue to have a "Prepared" notion of a resource, and will continue to provide the control over those states that it does today. Other applications may replace that completely with their own, or supplement it
Sounds good. I don't see any other critical issues so far then. The client code might get a bit complicated if one wants to abstract itself from specific resource managers implementations but flexibility always comes with a price tag. Let's see what the Ogre's team thinks about all this.
Sqeaky
Gnoblar
Posts: 24
Joined: Sat Jun 19, 2010 1:44 am

Re: Resource Manager

Post by Sqeaky »

Earlier threads were mentioned. Wherever threads are used it would be nice for the game developer to have an option on how they get used.

Clearly just having a threadpool do loading per thread is not efficient, most of that time threads will be IO bound and the cpu will sit idle. Some developers will not want to deal with any complexity and will accept longer load screens rather than accepting any complexity (real or percieved) that using threads would incur.
Need an alternative to a single threaded main loop for a game: https://github.com/BlackToppStudios/DAGFrameScheduler/
--Sqeaky
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

As Klaim asked for earlier; a (quickly created) collaboration diagram of how I expect the resource system to be split after this change:

Image

(It's only now I've finished creating this that I realise that Google Docs Draw has a snap to grid feature... sigh)

The dotted line around ResourceGroupManager/etc is to imply "ResourceLoader will be implemented /somewhere/ in there, and that the Mesh/etc loader will also be used /somewhere/ in there, but where is undecided yet :-)

---

My current opinion is that there should be one ResourceLoader instance per Resource. The loader will need some form of per resource data, and either you store two pointers in the resource (one to the loader, one to the data), or you store the data in the resource loader (in which case one pointer to the loader in the resource, one vtable pointer per loader).

Performance and memory impact is probably minimal (when compared to meshes which are at least a few KB, and textures which range into the MB, and considering this should be rarely executed code)
Sqeaky wrote:Earlier threads were mentioned. Wherever threads are used it would be nice for the game developer to have an option on how they get used.

Clearly just having a threadpool do loading per thread is not efficient, most of that time threads will be IO bound and the cpu will sit idle. Some developers will not want to deal with any complexity and will accept longer load screens rather than accepting any complexity (real or percieved) that using threads would incur.
OgreMain would no longer use any threads for resource loading; it would ask the loader to load the resource, and it might do it synchronously or asynchronously. Either way, OgeeMain is entirely decoupled. The default resource loader will continue to, as now, load resources "synchronously" in a pool of threads; a game which desires otherwise can implement threading as it desires
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: Resource Manager

Post by Klaim »

Some comments on the graphic:

1. I think the arrow between ResourceGroupManager and ResourceLoader is reversed: it's ResourceGroupManager which implements ResourceLoader.
2. If Mesh "uses" Resource, then Mesh doesn't inherit from resource? Maybe you should use the "implements" arrow for these.
3. I'm assuming the 3 parts on the left would be components, not plugins. Am I correct?

Otherwise it's clear. Maybe it lacks mention of shared pointers for resource too, and where they are used (just one or two examples would make it clear). But really it's clear enough with these data.
Owen wrote:My current opinion is that there should be one ResourceLoader instance per Resource.
Did you mean "one ResourceLoader instance per Resource child type"? Or did you mean "per Resource instance"?
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

Klaim wrote:Some comments on the graphic:

1. I think the arrow between ResourceGroupManager and ResourceLoader is reversed: it's ResourceGroupManager which implements ResourceLoader.
Correct. Derp.
Klaim wrote: 2. If Mesh "uses" Resource, then Mesh doesn't inherit from resource? Maybe you should use the "implements" arrow for these.
Mesh "extends" Resource (as now)
Klaim wrote: 3. I'm assuming the 3 parts on the left would be components, not plugins. Am I correct?
Correct. They will each be an (optional) library.
Klaim wrote:Otherwise it's clear. Maybe it lacks mention of shared pointers for resource too, and where they are used (just one or two examples would make it clear). But really it's clear enough with these data.
At least at first I'm thinking of staying with SharedPtr as now for resources. Longer term, I'd like to consider moving to an intrusive smart pointer which gives the loader more information and more control (in particular, the loader might place the resource in a cache at refcnt=0; it is the loader's policy which controls when the resource object actually gets unloaded and/or destroyed). The reference count would act as information for the loader
Klaim wrote:
Owen wrote:My current opinion is that there should be one ResourceLoader instance per Resource.
Did you mean "one ResourceLoader instance per Resource child type"? Or did you mean "per Resource instance"?
Per Resource instance.
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: Resource Manager

Post by Klaim »

Owen wrote: Did you mean "one ResourceLoader instance per Resource child type"? Or did you mean "per Resource instance"?
Per Resource instance.[/quote]

Then what would be the lifetime of a ResourceLoader instance? the whole Resource instance lifetime? Or will it be deleted as soon as unnecessary?
I know it looks like a minor details but I think depending on what is setup it that might greatly impact the memory behavior of applications using Ogre.
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

Klaim wrote:Then what would be the lifetime of a ResourceLoader instance? the whole Resource instance lifetime? Or will it be deleted as soon as unnecessary?
I know it looks like a minor details but I think depending on what is setup it that might greatly impact the memory behavior of applications using Ogre.
The ResourceLoader must stick around for at least as long as the Resource exists (its' responsible for reloading it in case of e.g. device loss, and control of the Resource's lifetime is delegated to it). Beyond that, the lifespan of the ResourceLoader is dictated by the application/resource manager.

----

I worked through what changes I imagine making to the Resource class.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Resource Manager

Post by dark_sylinc »

I'm going to write a more elaborate answer later, but I wanted to comment on this quote, because there's an inminent GSoC:
Owen wrote: All parts of Ogre which at present reference named resources will be converted to just use DataStreams.
That is a very bad idea. Removing strings is good, but DataStreams are flawed in various ways. For this particular case, file DataStream keep the File handle open until it's destruction. Since there is a limit on how many file handles/descriptors can remain opened at the same time by one process (in Windows the limit is 512) such code is destined to crash.

We had that problem with OgreOgg in The Dead Linger, OgreOgg was caching data using DataStream and eventually we ran out of file descriptors, worst part was that the game didn't crash until we tried to open a Mesh file; so the crash was very far apart from the actual source of failure.
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

dark_sylinc wrote:I'm going to write a more elaborate answer later, but I wanted to comment on this quote, because there's an inminent GSoC:
Owen wrote: All parts of Ogre which at present reference named resources will be converted to just use DataStreams.
That is a very bad idea. Removing strings is good, but DataStreams are flawed in various ways. For this particular case, file DataStream keep the File handle open until it's destruction. Since there is a limit on how many file handles/descriptors can remain opened at the same time by one process (in Windows the limit is 512) such code is destined to crash.

We had that problem with OgreOgg in The Dead Linger, OgreOgg was caching data using DataStream and eventually we ran out of file descriptors, worst part was that the game didn't crash until we tried to open a Mesh file; so the crash was very far apart from the actual source of failure.
Sorry, I should have elaborated on that earlier: I meant that only for the case of functions which take a reference to a file to access via the resource system (but don't access a resource), such as SceneManager->setWorldGeometry; I.E. this is for any code which currently accepts a name and passes it to ResourceGroupManager::openResource. I don't think Ogre has any code which does this at present which doesn't also feature a DataStream(Ptr) override; I merely included it as a final "catch-all" case.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Resource Manager

Post by dark_sylinc »

OK, So here's my view on the take. I will first admit that I haven't thought an entire Resource System refactoring thoroughly. I do know, that if I were to do it, I wouldn't refactor, just ditch the entire system (but keep the ScriptParsers). However there are a "bare minimum" elements that I feel need to be addressed:

Features
* Automatic Subcategories: For example in Python folders are namespaces. So what is in Linux "Folder/MyMaterials/DiffuseMats.material White" (White is defined in material DiffuseMats.material) in Windows would be located in "Folder\MyMaterials\DiffuseMats.material White".
Python both are treated as: "Folder.MyMaterials.DiffuseMats.White"
This way everything is non-ambiguous and there would no longer a need for resource group names (a largely unused feature in the current system). If the target architecture has a different way of representing files (i.e. a file system without folders, or a ZIP format, a TAR archive, etc etc) then it's converted to namespaces following predefined rules.
I believe Python has a few more rules (it's not just "change backlashes for dots") and it's worth looking at.

* Creation is initialization: D3D10 requires that static content to be sent when it is being created. In D3D9 & GL you could create the buffer, lock once, and send the data. Ogre follows the latter idiom and hence until we don't change it, propper D3D10 support will be a PITA, because we can't ever lock static buffers.

Multithreading
I believe there is no need to use a ResourceLoader per Resource. That's too wasteful. When designing multithreaded stuff that is supposed to load in the background, there are a few things to look for:

* Resource creation & destruction happens outside frame rendering: This is basic multithreaded design. Don't add or remove a Material mid frame. You can query to create/destroy after frame update finished tough.

* Blocking loads vs query loads: Instead of having one ResourceLoader per Resource, enforce that the Resource cannot be touched until it is loaded. How can this be possible? Easy, Resources have these two states: Loaded & Loading. The developer knows some resources are of vital importance, while others are not.

Suppose the game request with enough anticipation the creation of two meshes: "MainVillain.mesh" and "NotImportantTinyRock.mesh". Of course the first is of absolute importance, the second one is not. By "enough anticipation" I mean the game knows the player is entering the boss area, so it's pretty sure he may enconter the main villain, and hence queries the creation.

When the villain appears, the game will request the Mesh ptr (Mesh villain = MeshManager::getLoadedResource( mainVillainHandle, bool block = true ). Because block == true, the thread will be locked until the background thread signals the main one that MainVillain.mesh has been loaded. Therefore the main thread continues execution and draws the mesh. In other words, if the mesh hasn't been loaded yet, serialize it (a small hiccup could be noticeable).

And what about "NotImportantTinyRock.mesh"? It is a prop placed around the boss area. It's not really important. When getLoadedResource is called and sees the mesh isn't ready, it simply returns a null pointer. The game is allowed not to render the mesh this frame (try again in the next one) or replace it by a placeholder (this placeholder must be guaranteed to have been loaded, i.e. at startup during the "Loading screen")
Of course the ResourceSystem interface can do this automatically: Return the place holder instead of a null pointer and an output reference to indicate it's the predefined placeholder.
In code:

Code: Select all

enum LoadingReturnStatus
{
 LOADED, //Resource loaded successfully
 LOADED_SERIALIZED, //Resource loaded successfully, but we had to wait.
 STILL_LOADING_NULL, //Resource is still being loaded, returned null ptr
 STILL_LOADING_PLACEHOLDER, //Resource is still being loaded, returned the placeholder instead
};

ResourceId createResource( ... );
Resource* getLoadedResource( ResourceId resourceBeingLoaded, bool block, ResourceId placeHolderAlternative, LoadingReturnStatus &outLoadingStatus );
This way there is no need to have multiple ResourceLoader that duplicate the data present in the Resource. It's all in the resource. You may have noticed the pointer is not available to us until getLoadedResource, all we got in createResource() was a "ticket" so we can claim the pointer when it's ready.
To handle resource reloading, we just query the reload, and while it is reloading, we promise not to touch the Resource pointer. We can even enforce it by making the input of two indirections as in "ResourceId Manager::reload( Resource **ptr )" so that we set *ptr to null and the user can reclaim it with the ticket we gave (it's unique id).

Of course, all of this is highly incompatible with the current system, so I haven't thought how it would be replaced; which I think is the toughest part.
Additionally "resource.cfg" should be "self-explained" so that it contains all important info a DCC tools need (making import/export process as straightforward and fast as possible). For example, inside there could be stored which Resources can be replaced by which placeholder, and which Resources need to be blocking (and the DCC tool can edit it)
Other info needed by DCC tools could be which folders are for meshes (so that artist just hits 'save'), or name filters (i.e. everything ending with "tree" and "leaves" goes into the "Tree" directory). While this info may or may not be used by the game, at least all DCC tools remain consistent.

Cheers
Dark Sylinc
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

dark_sylinc wrote:OK, So here's my view on the take. I will first admit that I haven't thought an entire Resource System refactoring thoroughly. I do know, that if I were to do it, I wouldn't refactor, just ditch the entire system (but keep the ScriptParsers). However there are a "bare minimum" elements that I feel need to be addressed:
My first intention is to take the resource system and completely remove it from OgreMain. Gone. For good. Never coming back.

Good riddance. My app doesn't need it or want it, and can't live with it.

Script parsers are going in their own library. Mesh loader is going in its own library. Going to extend the codecs system already used for images to all resource types, at least where applicable.
dark_sylinc wrote:Features
* Automatic Subcategories: For example in Python folders are namespaces. So what is in Linux "Folder/MyMaterials/DiffuseMats.material White" (White is defined in material DiffuseMats.material) in Windows would be located in "Folder\MyMaterials\DiffuseMats.material White".
Python both are treated as: "Folder.MyMaterials.DiffuseMats.White"
This way everything is non-ambiguous and there would no longer a need for resource group names (a largely unused feature in the current system). If the target architecture has a different way of representing files (i.e. a file system without folders, or a ZIP format, a TAR archive, etc etc) then it's converted to namespaces following predefined rules.
I believe Python has a few more rules (it's not just "change backlashes for dots") and it's worth looking at.
Resource groups are a feature absolutely orthogonal to the naming of resources (in particular, they enable force loading/unloading of a collection of resources). Regardless, this is all irrelevant to the core of my proposed changes (i.e. how the "default" resource system should look is another matter)
dark_sylinc wrote: * Creation is initialization: D3D10 requires that static content to be sent when it is being created. In D3D9 & GL you could create the buffer, lock once, and send the data. Ogre follows the latter idiom and hence until we don't change it, propper D3D10 support will be a PITA, because we can't ever lock static buffers.
Certainly that should be done for buffers. Whether it is in scope for this project is another matter (Perhaps this is better rolled in with a project to finish the D3D11 render system?)
dark_sylinc wrote: Multithreading
I believe there is no need to use a ResourceLoader per Resource. That's too wasteful.
For every resource, Ogre needs a way to link it to the resource system that loaded it.

Picture an application which uses its' own resource system and CEGUI. CEGUI should not, for the purposes of interacting with Ogre, need to know anything about the application's resource system. Instead, the idea is that it can create the Resources directly and provide it's own ResourceLoader for them (like, if memory serves, it provides a ManualResourceLoader now).

So, a resource needs a link to its' loader, and that loader likely needs some way to identify the resource (being as, as I've mentioned, resource names/IDs are on my chopping block)

We could put a void*/Orge::Any in the Resource, and let the loader use that... but to me it feels simpler and more elegant for the resource system to just store that information in the ResourceLoader instance

Think of it as every resource being loaded by a ManualResourceLoader. Thats the paradigm I want to head to.
dark_sylinc wrote: When designing multithreaded stuff that is supposed to load in the background, there are a few things to look for:

* Resource creation & destruction happens outside frame rendering: This is basic multithreaded design. Don't add or remove a Material mid frame. You can query to create/destroy after frame update finished tough.
Agreed. As mentioned, I'm making Resource completely thread unsafe except that
  • The reference count will be maintained atomically
  • The main thread guarantees to update the resource state atomically (i.e. it is legal to read it from other threads)
dark_sylinc wrote: * Blocking loads vs query loads: Instead of having one ResourceLoader per Resource, enforce that the Resource cannot be touched until it is loaded. How can this be possible? Easy, Resources have these two states: Loaded & Loading. The developer knows some resources are of vital importance, while others are not.

Suppose the game request with enough anticipation the creation of two meshes: "MainVillain.mesh" and "NotImportantTinyRock.mesh". Of course the first is of absolute importance, the second one is not. By "enough anticipation" I mean the game knows the player is entering the boss area, so it's pretty sure he may enconter the main villain, and hence queries the creation.

When the villain appears, the game will request the Mesh ptr (Mesh villain = MeshManager::getLoadedResource( mainVillainHandle, bool block = true ). Because block == true, the thread will be locked until the background thread signals the main one that MainVillain.mesh has been loaded. Therefore the main thread continues execution and draws the mesh. In other words, if the mesh hasn't been loaded yet, serialize it (a small hiccup could be noticeable).

And what about "NotImportantTinyRock.mesh"? It is a prop placed around the boss area. It's not really important. When getLoadedResource is called and sees the mesh isn't ready, it simply returns a null pointer. The game is allowed not to render the mesh this frame (try again in the next one) or replace it by a placeholder (this placeholder must be guaranteed to have been loaded, i.e. at startup during the "Loading screen")
Of course the ResourceSystem interface can do this automatically: Return the place holder instead of a null pointer and an output reference to indicate it's the predefined placeholder.
In code:

Code: Select all

enum LoadingReturnStatus
{
 LOADED, //Resource loaded successfully
 LOADED_SERIALIZED, //Resource loaded successfully, but we had to wait.
 STILL_LOADING_NULL, //Resource is still being loaded, returned null ptr
 STILL_LOADING_PLACEHOLDER, //Resource is still being loaded, returned the placeholder instead
};

ResourceId createResource( ... );
Resource* getLoadedResource( ResourceId resourceBeingLoaded, bool block, ResourceId placeHolderAlternative, LoadingReturnStatus &outLoadingStatus );
This way there is no need to have multiple ResourceLoader that duplicate the data present in the Resource. It's all in the resource. You may have noticed the pointer is not available to us until getLoadedResource, all we got in createResource() was a "ticket" so we can claim the pointer when it's ready.
To handle resource reloading, we just query the reload, and while it is reloading, we promise not to touch the Resource pointer. We can even enforce it by making the input of two indirections as in "ResourceId Manager::reload( Resource **ptr )" so that we set *ptr to null and the user can reclaim it with the ticket we gave (it's unique id).
This approach is unworkable - it would require to traverse all Entities and similar whenever the render system drops the device on top of us, in order to erase their resource references (and then instigate reloading of them)

Ogre is a retained mode graphics API. Resources should be retained also.

Additionally, I don't like the idea of keeping blocking resource loads (For example, the application's resource system might rely upon the main thread servicing an event loop of some kind in order to keep the resource system functional)

Loading priorities is, of course, a feature that could (and should) be added to the default resource system.
dark_sylinc wrote:Of course, all of this is highly incompatible with the current system, so I haven't thought how it would be replaced; which I think is the toughest part.
Additionally "resource.cfg" should be "self-explained" so that it contains all important info a DCC tools need (making import/export process as straightforward and fast as possible). For example, inside there could be stored which Resources can be replaced by which placeholder, and which Resources need to be blocking (and the DCC tool can edit it)
Other info needed by DCC tools could be which folders are for meshes (so that artist just hits 'save'), or name filters (i.e. everything ending with "tree" and "leaves" goes into the "Tree" directory). While this info may or may not be used by the game, at least all DCC tools remain consistent.
One of the resultant benefits of my proposed excision of the resource system is that innovation in the area becomes much easier (i.e. once can just implement their desired resource manager, and experimentation should be easier)
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Resource Manager

Post by dark_sylinc »

Owen wrote:For every resource, Ogre needs a way to link it to the resource system that loaded it.

Picture an application which uses its' own resource system and CEGUI. CEGUI should not, for the purposes of interacting with Ogre, need to know anything about the application's resource system. Instead, the idea is that it can create the Resources directly and provide it's own ResourceLoader for them (like, if memory serves, it provides a ManualResourceLoader now).

So, a resource needs a link to its' loader, and that loader likely needs some way to identify the resource (being as, as I've mentioned, resource names/IDs are on my chopping block)

We could put a void*/Orge::Any in the Resource, and let the loader use that... but to me it feels simpler and more elegant for the resource system to just store that information in the ResourceLoader instance

Think of it as every resource being loaded by a ManualResourceLoader. Thats the paradigm I want to head to.
I don't see how the description I made of the system would be incompatible of what you're describing.

There would be a ResourceLoader provided by CEGUI, just it's not per Resource instance, it's global. The ResourceSystem still needs to know how to fill the resource, and that's why a custom ResourceLoader is needed.

As for storing the needed data as void*/Ogre::Any; I don't get it. If CEGUI needs custom resources, then deriving from the base Resource is the right solution:

CEGUITextureLoader : public Ogre::ResourceLoader ---> Handles filling of Resources.
CEGUITexture : public Ogre::Texture ---> CEGUI's texture, data handled & filled by CEGUITextureLoader
Owen wrote: Agreed. As mentioned, I'm making Resource completely thread unsafe except that
  • The reference count will be maintained atomically
  • The main thread guarantees to update the resource state atomically (i.e. it is legal to read it from other threads)
For the first one, some day Ogre will have a defined way of ditching resources and reference count will hopefully go away (or stay in a way more like Havok does). But not today, so, acceptable.
Could you elaborate on the latter (what do you mean to be legal to read the state from other threads, and why the main thread would be the one updating the resource state)?
Owen wrote:This approach is unworkable - it would require to traverse all Entities and similar whenever the render system drops the device on top of us, in order to erase their resource references (and then instigate reloading of them)
1. Render Systems don't drop the device often.
2. Even if it did, the system would still handle it gracefully. On device lost the resource system stalls everything else until everything is reloaded. This approach is serialized, but it's pointless to talk about "background loading" when there's nothing in the foreground to do.
Owen wrote:Ogre is a retained mode graphics API. Resources should be retained also.
When I'm talking about releasing the Resource pointer, I'm not saying that it will become a dangling ptr. The same pointer will be delivered when the resource is given back.
The reason I'm saying we should encourage to release the Resource is because it's multithreaded (aka. avoid the user from screwing up, he's not expected to know much about MT safety or how the ResourceSys works); you can't use a resource that is not yet loaded or in an inconsistent state. You either wait (stall), or try again next time. If you choose the latter, it's safer if we take away the dangerous toy.

This is made for the sake of multithreading, scalability, and safety. If you wish for all sakes to retain the pointer, use blocking loads & reloads for all your Resources. With enough anticipation, blocking loads will never stall; though I admit for reloads it would always stall unless you give away the Resource for a while and do something else until is reloaded. Nevertheless any other multithreading aproach that avoids this problem means more frequent locks to check the state.

My approach needs to lock once per frame. Blocking loads may or may not need to lock once per call (they definitely don't if the resource was already loaded; if it wasn't it... depends, keep reading for an explanation)
Owen wrote: Additionally, I don't like the idea of keeping blocking resource loads (For example, the application's resource system might rely upon the main thread servicing an event loop of some kind in order to keep the resource system functional)
Well, I'm strongly inclined against anything that could lead to such kind of inter-dependency because it ends up in listeners pre- and post- anything you'll perform (I'm speaking generally, not just Resources) which leads to:
a. if( listener ) listener->callVirtualFunction() -> This is bad if done too often.
b. Break assumptions on the state of what you're working with, that can lead to preventing optimizations on how to treat the data you're processing -> The worst side effect. Node::processQueuedUpdates is a prime example of this misfortune.

And 99% times, there's a better way of doing it (sometimes, this way already existed but was ignored).

However, I do recognize there may be times where this is needed, may be because of time constraints, legal (i.e. not all source code is available), etc. which may render this inter-dependency necessary. So, shall that be the case, my answer is:

Don't take my words so literally. Making getLoadedResource 'blocking' doesn't necessarily mean the thread gets suspended. Sure that's probably what's often gonna happen. But what it really means is that we guarantee that getLoadedResource won't return until the pointer is loaded, it doesn't matter how.
In your particular example, one solution is that the thread gets suspended and waits for the Background to thread to wake him up to service the event loop you mentioned and then goes to sleep again. After the resource is loaded, the main thread wakes up again for a second time, this time to return from getLoadedResource.

That's one solution, another attempt is to flag all those cases and load the resource inside getLoadedResource (that is, no multithreading at all). I can think of other similar single threaded solutions.
Another multithreaded solution is that a task that will service the event loop is spawned before the main thread goes to sleep.
Owen wrote:One of the resultant benefits of my proposed excision of the resource system is that innovation
I'm looking forward to that.

Cheers
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

dark_sylinc wrote:I don't see how the description I made of the system would be incompatible of what you're describing.

There would be a ResourceLoader provided by CEGUI, just it's not per Resource instance, it's global. The ResourceSystem still needs to know how to fill the resource, and that's why a custom ResourceLoader is needed.

As for storing the needed data as void*/Ogre::Any; I don't get it. If CEGUI needs custom resources, then deriving from the base Resource is the right solution:

CEGUITextureLoader : public Ogre::ResourceLoader ---> Handles filling of Resources.
CEGUITexture : public Ogre::Texture ---> CEGUI's texture, data handled & filled by CEGUITextureLoader
For textures and shaders, Ogre has already taken the subclasses for its' own purposes. Additionally, the resource system would likely wish to be generic about the resource types its' handling.

Therefore, I decided that the ResourceLoader should take that responsibility.

Eseentially, we are just splitting the existing Resource down the middle, and placing half the state in Resource and its subclasses, and the other half in ResourceLoader's subclasses.
dark_sylinc wrote: For the first one, some day Ogre will have a defined way of ditching resources and reference count will hopefully go away (or stay in a way more like Havok does). But not today, so, acceptable.
Could you elaborate on the latter (what do you mean to be legal to read the state from other threads, and why the main thread would be the one updating the resource state)?
Regarding reference count: My intention was that the resource system would not hold a reference to the resource; therefore, unless the app is holding on to the Resource, it would be a direct proxy for whether the resource is in use or not. The loader would be notified when the reference count dropped to zero, and could then implement policy regarding when it unloads the resource. (That is, the reference count would not be directly related to the lifetime of the Resource)

By state I simply mean the "Unloaded/Loading/Loaded" state variable. Nothing else.

Resource will only really be concerned with Ogre's view of the resource, and almost all (if not all) resource types that Ogre deals with are deeply GPU related, and mostly this means that they can only be loaded on the main render thread. Formalizing that arrangement to Resource means that the thread safety of it can be simplified, and, importantly, from Ogre's perspective Resource can never be in an inconsistent state.

A threaded/asynchronous resource system would do most of its work on a background thread, and once it had its' data prepared, it would do the final stage of preparation on the main thread. Prior to that, all state changes are the confined to the ResourceLoader.
dark_sylinc wrote: 1. Render Systems don't drop the device often.
2. Even if it did, the system would still handle it gracefully. On device lost the resource system stalls everything else until everything is reloaded. This approach is serialized, but it's pointless to talk about "background loading" when there's nothing in the foreground to do.
Hmm, OK. I'm still slightly wary of that, however (Depending upon how many resources we are discussing, the app might want to throw up a loading screen)
dark_sylinc wrote: When I'm talking about releasing the Resource pointer, I'm not saying that it will become a dangling ptr. The same pointer will be delivered when the resource is given back.
The reason I'm saying we should encourage to release the Resource is because it's multithreaded (aka. avoid the user from screwing up, he's not expected to know much about MT safety or how the ResourceSys works); you can't use a resource that is not yet loaded or in an inconsistent state. You either wait (stall), or try again next time. If you choose the latter, it's safer if we take away the dangerous toy.

This is made for the sake of multithreading, scalability, and safety. If you wish for all sakes to retain the pointer, use blocking loads & reloads for all your Resources. With enough anticipation, blocking loads will never stall; though I admit for reloads it would always stall unless you give away the Resource for a while and do something else until is reloaded. Nevertheless any other multithreading aproach that avoids this problem means more frequent locks to check the state.

My approach needs to lock once per frame. Blocking loads may or may not need to lock once per call (they definitely don't if the resource was already loaded; if it wasn't it... depends, keep reading for an explanation)
As I've said, Resource is for all intents and purposes immutable except on the render thread. The only case where we might lose performance with my method is that a conditional would be needed for resources which haven't yet loaded, but then again Ogre could always fill those fields with values from a "fill-in" resource if that branch was an issue.

I'm not sure it will be. It certainly can't be worse than the indirect calls of WGL, vtable calls of DirectX or doubly indirect calls of GLX/AGL/EGL, particularly if most of the time it should be predicted successfully?
dark_sylinc wrote: Well, I'm strongly inclined against anything that could lead to such kind of inter-dependency because it ends up in listeners pre- and post- anything you'll perform (I'm speaking generally, not just Resources) which leads to:
a. if( listener ) listener->callVirtualFunction() -> This is bad if done too often.
b. Break assumptions on the state of what you're working with, that can lead to preventing optimizations on how to treat the data you're processing -> The worst side effect. Node::processQueuedUpdates is a prime example of this misfortune.

And 99% times, there's a better way of doing it (sometimes, this way already existed but was ignored).

However, I do recognize there may be times where this is needed, may be because of time constraints, legal (i.e. not all source code is available), etc. which may render this inter-dependency necessary. So, shall that be the case, my answer is:

Don't take my words so literally. Making getLoadedResource 'blocking' doesn't necessarily mean the thread gets suspended. Sure that's probably what's often gonna happen. But what it really means is that we guarantee that getLoadedResource won't return until the pointer is loaded, it doesn't matter how.
In your particular example, one solution is that the thread gets suspended and waits for the Background to thread to wake him up to service the event loop you mentioned and then goes to sleep again. After the resource is loaded, the main thread wakes up again for a second time, this time to return from getLoadedResource.

That's one solution, another attempt is to flag all those cases and load the resource inside getLoadedResource (that is, no multithreading at all). I can think of other similar single threaded solutions.
Another multithreaded solution is that a task that will service the event loop is spawned before the main thread goes to sleep.
Understood. We are wandering into the territory of what I'd describe as the "new default resource system" here, however; and blocking loads are something I want completely out of OgreMain (as said, I have a project which needs completely asynchronous resource loading; if we are downloading from a server with 300ms of latency, then it could quite easily be 1s before the resource can even start loading)
User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
x 13
Contact:

Re: Resource Manager

Post by sparkprime »

2 questions

1) Do we really believe that we can easily get good performance on all targets for async io at this point in time? Edit: libevent maybe?
2) I don't buy the idea that because you are downloading from a network rather than a local disk and therefore can't use blocking threads, that the whole implementation must not use blocking threads. You can always get the buffer yourself and hand it ogre afterwards.
User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
x 13
Contact:

Re: Resource Manager

Post by sparkprime »

dark_sylinc wrote: * Automatic Subcategories: For example in Python folders are namespaces. So what is in Linux "Folder/MyMaterials/DiffuseMats.material White" (White is defined in material DiffuseMats.material) in Windows would be located in "Folder\MyMaterials\DiffuseMats.material White".
Python both are treated as: "Folder.MyMaterials.DiffuseMats.White"
This way everything is non-ambiguous and there would no longer a need for resource group names (a largely unused feature in the current system). If the target architecture has a different way of representing files (i.e. a file system without folders, or a ZIP format, a TAR archive, etc etc) then it's converted to namespaces following predefined rules.
I believe Python has a few more rules (it's not just "change backlashes for dots") and it's worth looking at.
The way I handled this in the Grit engine is to have resource references be local to the resource that is using them. E.g. if you have a material Blah defined in a script in dir 'foo' then the material that you are defining is actually called '/foo/Blah'. If you have /foo/MyMesh.mesh which refers to material Blah, then the material used is actually /foo/Blah. This allows avoiding name clashes and easy moving of stuff around directories since all the paths are typically local. You can also put in absolute paths or use .. to go down a dir.

I don't use any resource groups or any crap like that.
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

sparkprime wrote:2 questions

1) Do we really believe that we can easily get good performance on all targets for async io at this point in time? Edit: libevent maybe?
A simple select() loop is sufficient for the level of parallelism expected from reasonable applications (~10 parallel connections). libevent and similar are great and very useful, but mostly unnecessary outside of servers
sparkprime wrote:2) I don't buy the idea that because you are downloading from a network rather than a local disk and therefore can't use blocking threads, that the whole implementation must not use blocking threads. You can always get the buffer yourself and hand it ogre afterwards.
Essentially, by separating the resource manager as outlined I'm making Ogre entirely ambivalent to the method used to load the resource. Ogre just asks the system to load the resource and doesn't care whether it is done synchronously, asynchronously, in a background thread, etc.
User avatar
Zonder
Ogre Magi
Posts: 1168
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 73

Re: Resource Manager

Post by Zonder »

Owen wrote:
sparkprime wrote:2 questions

1) Do we really believe that we can easily get good performance on all targets for async io at this point in time? Edit: libevent maybe?
A simple select() loop is sufficient for the level of parallelism expected from reasonable applications (~10 parallel connections). libevent and similar are great and very useful, but mostly unnecessary outside of servers
sparkprime wrote:2) I don't buy the idea that because you are downloading from a network rather than a local disk and therefore can't use blocking threads, that the whole implementation must not use blocking threads. You can always get the buffer yourself and hand it ogre afterwards.
Essentially, by separating the resource manager as outlined I'm making Ogre entirely ambivalent to the method used to load the resource. Ogre just asks the system to load the resource and doesn't care whether it is done synchronously, asynchronously, in a background thread, etc.
I think ogre should always expect something to load async although it might not be in the resource manager, it gives flexibility then. Ogre itself should use placeholders also for resources that can't be found or waiting to load regardless if they are loading async/sync so if a resource is loading async rendering just continues but when the resource becomes available the place holder is no longer used. In a filesystem based resource manager you could use error place holders / web based you could use blank place holders for example.
There are 10 types of people in the world: Those who understand binary, and those who don't...
User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
x 13
Contact:

Re: Resource Manager

Post by sparkprime »

Owen wrote:
sparkprime wrote:2 questions

1) Do we really believe that we can easily get good performance on all targets for async io at this point in time? Edit: libevent maybe?
A simple select() loop is sufficient for the level of parallelism expected from reasonable applications (~10 parallel connections). libevent and similar are great and very useful, but mostly unnecessary outside of servers
It's not select() though, it's the fact that doing non-blocking I/O e.g. on hard disk access actually results in a blocking call, thus making the whole design malfunction.

You may find you have non-blocking for IPC pipes and network I/O but not on hard disk I/O.

I believe this is why the current implementation uses threads.
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: Resource Manager

Post by Owen »

sparkprime wrote:It's not select() though, it's the fact that doing non-blocking I/O e.g. on hard disk access actually results in a blocking call, thus making the whole design malfunction.

You may find you have non-blocking for IPC pipes and network I/O but not on hard disk I/O.

I believe this is why the current implementation uses threads.
The whole point is that Ogre shouldn't care about how the manager loads the resource. The default ResourceManager will continue to use threads and mostly behave as it does now.
User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
x 13
Contact:

Re: Resource Manager

Post by sparkprime »

OK so if you generalise the Ogre API to allow apps to use async disk I/O where that is available, that seems pretty useful. I may even use it myself :)
Post Reply