Page 1 of 1

How to determine and discard unused HLMS textures

Posted: Sat Nov 18, 2017 2:35 am
by crancran
In a streaming world simulation, parts of the world are constantly being loaded and unloaded. If we consider the terrain, its often split into a grid structure where pages are loaded/unloaded based on a view distance. But pages often share textures among them as multiple terrain pages can make up a similar themed area like a dessert, forest, etc.

As a terrain page gets loaded, I iterate the textures used by the entire page and preemptively load them into the Hlms texture atlas by:

Code: Select all

hlmsTextureManager->createOrRetrieveTexture( textureName, TEXTURE_TYPE_DIFFUSE );
Later as a I am building the HlmsDataBlock instances required for the terrain cells (smaller grid rendered areas of the page), I get the TextureLocation from the Hlms TextureManager and use that to build the data block for the materials.

Building up the materials needed to render the pages & cells is pretty straight forward and simple.

During the unload phase of a terrain page, I need to be able to determine whether or not a particular texture that was used by the page is being used by any other pages and if not, unload it from the Hlms Texture Manager/Atlas.

Does OGRE offer any clean way for me to determine whether that texture is no longer used and discard it without having to resort to writing my own reference management for doing this?

Re: How to determine and discard unused HLMS textures

Posted: Sun Nov 19, 2017 8:45 pm
by dark_sylinc

First, Texture management is unfortunately Ogre 2.1's weakest spot. Which is what 2.2 will be addressing.
For example, an Hlms material, even if not used, will load the textures immediately (which is wasteful). You would have to create the materials and the texture with the page.

Ogre 2.2 also has background streaming for loading textures. Unfrotunately 2.2 is WIP. You could try it but I cannot guarantee everything will work as expected.

Now, onto your specific question:
Each datablock tracks what Renderables are using it. This is in HlmsDatablock::getLinkedRenderables.
If multiple materials are using the same texture, we don't have a way of knowing easily (you could iterate through all datablocks from each Ogre::Hlms and check one by one though...). But you can be certain that if getLinkedRenderables returns an empty array, then that material can be destroyed. If you somehow can determine that the textures associated by that material are also not being used, then you could remove them too.
Ogre 2.2 does have a way of knowing, as in a similar fashion to HlmsDatablocks, TextureGpu can have listeners (TextureGpu::mListeners). Each material/datablock adds itself to the TextureGpu as a listener, so if a TextureGpu has no listeners, then it's not being used anywhere (and if it is and crashes, then that's likely a bug, as the piece of code keeping a reference to the TextureGpu pointer should be listening for changes).

Re: How to determine and discard unused HLMS textures

Posted: Fri Nov 24, 2017 11:07 pm
by crancran
Is there any special toggles in the build that should be enabled for this texture streaming functionality or is it just baked in by default in 2.2?

Re: How to determine and discard unused HLMS textures

Posted: Sat Nov 25, 2017 12:12 am
by dark_sylinc
It's forced on by default.

There's a "#define OGRE_FORCE_TEXTURE_STREAMING_ON_MAIN_THREAD 1" in OgreTextureGpuManager.cpp to force single threaded, but it's only meant for debugging.
Users who want to wait for results (so that all frames are rendered perfectly instead of showing a dummy texture while we haven't finished loading) can use textureGpuManager->waitForStreamingCompletion which will stall until all pending textures are loaded.

The issue only right now with 2.2 is that while the texture hasn't loaded yet, a shader is used; and when the texture finishes loading a different shader is likely going to be used (causing stalls if the new shader wasn't cached and must be compiled). The details are explained in this post: viewtopic.php?f=25&t=89127&start=25#p537823 and the solution to that will be (not yet coded) a metadata cache that tells us in advance information about the texture (namely we need resolution and pixel format in advance to prevent that shader hiccup).