[2.1] Defintively there's no way to do asynchronous loading?

A place for users of OGRE to discuss ideas and experiences of utilitising OGRE in their games / demos / applications.
Post Reply
hydexon
Gremlin
Posts: 164
Joined: Sun Apr 14, 2013 8:51 pm
x 10

[2.1] Defintively there's no way to do asynchronous loading?

Post by hydexon »

So in earlier versions of OGRE has some kind of "half-working" background resource loading, sometimes it worked for someone, but somethimes not for others.

What i'm trying to do is achieve is asyncronous or background resource loading on my proper engine and later to be usable with OGRE, i already know OGRE isn't much friendly in multithreaded environments, i was thinking about roll my own Asset System (with Background Loading of course) or simply live with the OGRE's ResourceManager beast, already know how to load assets from raw data pointers without needing the resource system loading calls.

Also, i've already readed the 2012 GSoC Resource System refactor proposal which never made to the OGRE codebase so please don't post that link already.

At least i want to know if people their ways to overcome this, since i can't give up because OGRE is the most and fully-featured rendering engine which just meets the needs for our project.
So, the question is: in OGRE 2.1 there's an way to load resources in an background thread, with or without OGRE Resource System data loading mechanisms?.

Thanks you.-
limesX
Gnoblar
Posts: 17
Joined: Mon Nov 12, 2012 3:05 pm
x 2

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by limesX »

Hi.
Have you ever managed to achieve background loading with ogre's resource management system?
loath
Platinum Sponsor
Platinum Sponsor
Posts: 290
Joined: Tue Jan 17, 2012 5:18 am
x 67

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by loath »

i do asynchronous loading on 1.11.5 with dx9 and hlsl. is there anything that would prevent me from doing the same on the 2.1/2.2 code base?

here is what i currently do and it works very well for me:

i have a large worlds broken into chunks. as the player moves around i pre-load the next set of chunks. this includes loading all of the materials (textures, meshes, skeletons) and compiling all my shaders on my own background worker threads. i don't explicitly use any fixed function behaviors and i don't call any dx9 APIs (through the dx9 render system) on the worker threads.

to accomplish this i had to make a minor modification to vanilla ogre to make Ogre::HighLevelGpuProgram::loadHighLevel() public instead of private/protected. for dx9 this function essentially calls directx's D3DXCompileShader() which *can* be used on background threads. (unlike everything else in dx9).

once the materials are loaded and compiled, i post them to a worker queue that is picked up by the rendering thread. the rendering thread will load these raw materials into Ogre::Resource based objects over many frames to avoid framerate hiccups. when all of the resources are ready i allow the chunk to be rendered. i also do the same to unload all of the materials from chunks that are no longer needed.

i'd be curious to know if 2.1/2.2 will allow this. thanks!
al2950
OGRE Expert User
OGRE Expert User
Posts: 1227
Joined: Thu Dec 11, 2008 7:56 pm
Location: Bristol, UK
x 157

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by al2950 »

I can not answer all your questions, but with Ogre 2.2, it fully supports background loading and streaming of textures, in fact it just does it, you dont need to do anything special. To avoid other frame hiccups, you will also need to use shader and material caches properly, which is pretty easy to do, but is a different approach to dynamically compiling them in a background thread. So the only real problem area will be the loading of Mesh files , if you can't afford to load all meshes at start time then it will probably be a little harder to do that in a background thread in Ogre 2.2+. I believe background loading, at least as it was in 1.x, is not supported at all anymore. But I might be wrong.

You may also find it harder to implement your background loading because Ogre 2.1+ tries to combine multiple buffers (eg meshes) into a single buffer for performance reasons. So you need to be aware of this if you do your own loading, as you may hinder Ogre's 2.1+ ability to render things really fast :)

Hopefully dark_sylinc might be able to chime in to fill in any gaps I have left.
limesX
Gnoblar
Posts: 17
Joined: Mon Nov 12, 2012 3:05 pm
x 2

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by limesX »

In my older project (cannot find the code at the moment, but will look it up), based on Ogre 1.9 I managed to make this happen without any problems.
Don't know about hiccups because I had a scene with some text on it while everything was loading in the background. I was able to load materials, meshes and skeletons. All was done in the background. After everything loaded, I added entities in the main thread. Worked fine for me.
But know I have some problems. I made a new topic if you can help out.

viewtopic.php?f=5&t=95461

Thanks.
loath
Platinum Sponsor
Platinum Sponsor
Posts: 290
Joined: Tue Jan 17, 2012 5:18 am
x 67

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by loath »

al2950 wrote: Mon Oct 28, 2019 12:12 pm You may also find it harder to implement your background loading because Ogre 2.1+ tries to combine multiple buffers (eg meshes) into a single buffer for performance reasons. So you need to be aware of this if you do your own loading, as you may hinder Ogre's 2.1+ ability to render things really fast :)
thanks for the response al2950. i would prefer to keep asset resources combined as you mention. for example, a world chunk might have all of it's buffers combined for locality. the only exception would be that some chunks share terrain textures or say trees / props in order to transition artistically.

i should also mention that my worlds are procedurally generated which is why i compile all my shaders on the fly as chunks are loaded. i suppose an alternate for me to consider would be to compile the shaders when the world is generated (stored in the cache you mention) rather than on the fly.

thanks again!
loath
Platinum Sponsor
Platinum Sponsor
Posts: 290
Joined: Tue Jan 17, 2012 5:18 am
x 67

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by loath »

limesX wrote: Mon Oct 28, 2019 5:50 pm In my older project (cannot find the code at the moment, but will look it up), based on Ogre 1.9 I managed to make this happen without any problems.
Don't know about hiccups because I had a scene with some text on it while everything was loading in the background. I was able to load materials, meshes and skeletons. All was done in the background. After everything loaded, I added entities in the main thread. Worked fine for me.
But know I have some problems. I made a new topic if you can help out.

viewtopic.php?f=5&t=95461

Thanks.
background loading works great on 1.x but with shaders (which can take 300+ ms to compile), skeletons, non-trivial materials, etc you can get frame hicups if you don't "trickle" them onto the render thread and do what you can on your own background threads. (especially in my case where i load an entire chunk's worth of assets at a time).

edit: looks like dx11 shaders can take 5 seconds to compile. from here: http://wiki.ogre3d.org/tiki-index.php?p ... _resources_
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by dark_sylinc »

These questions come up every once in a while in the forums: yes, doing async loading in 2.1 can be very hard.
  1. Meshes are loaded synchronously. User 0xC0DEFACE was able to use the NULLRenderSystem to trick Ogre into loading meshes in the background using the NULLVaoManager, then perform a final copy to the real D3D11/GL RenderSystem. But certainly not something we're providing out of the box, and probably should not be 'user friendly'.
  2. Asynchronous loading of textures started in Ogre 2.2; in fact in 2.1 texture loading is very synchronous and needs to often load sooner rather than later. In Ogre 2.2 all texture loads are asynchronous by default.
  3. As the FAQ said, Hlms' shaders in D3D11 can take a while to compile, which is why we implemented the HlmsDiskCache and the microcode cache (compile once, reuse afterwards)
loath
Platinum Sponsor
Platinum Sponsor
Posts: 290
Joined: Tue Jan 17, 2012 5:18 am
x 67

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by loath »

thanks for the info and clarifications dark_sylinc!
User avatar
TaaTT4
OGRE Contributor
OGRE Contributor
Posts: 267
Joined: Wed Apr 23, 2014 3:49 pm
Location: Bologna, Italy
x 75
Contact:

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by TaaTT4 »

dark_sylinc wrote: Tue Oct 29, 2019 12:42 am Meshes are loaded synchronously. User 0xC0DEFACE was able to use the NULLRenderSystem to trick Ogre into loading meshes in the background using the NULLVaoManager, then perform a final copy to the real D3D11/GL RenderSystem. But certainly not something we're providing out of the box, and probably should not be 'user friendly'.
If I remember well, 0xC0DEFACE had very particular needs. And as Matias said, its trick was very far from being for beginner users.
A simple thing you can try is to build OGRE with OGRE_CONFIG_THREADS=2 and do asynchronous the resource preparation step (this phase happens when a resource transit from unload to load state). All this thing doesn't magically work out of the box, but you have to use the ResourceBackgroundQueue and ResourceBackgroundQueue::Listener classes.
Please note that the highest hiccups don't happen in loading meshes from disk, but when you create Item objects. A very effective trick is to span the items creation over multiple frames. Eg, if you have to create 10000 items, do it in 100 frames (100 items at a time).

Senior programmer at 505 Games; former senior engine programmer at Sandbox Games
Worked on: Racecraft EsportRacecraft Coin-Op, Victory: The Age of Racing

limesX
Gnoblar
Posts: 17
Joined: Mon Nov 12, 2012 3:05 pm
x 2

Re: [2.1] Defintively there's no way to do asynchronous loading?

Post by limesX »

Thanks everyone for tips and tricks. But I think it will be easier for me to change the concept of loading levels then to continue the battle with this background loading.....
Post Reply