Fastest way to load large number of textures Topic is solved

Problems building or running the engine, queries about how to use features etc.
Post Reply
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Fastest way to load large number of textures

Post by Oogst »

Ogre Version: 1.12.11
Operating System: Windows 10
Render System: DirectX 11

I have 913 DDS DXT1 textures, totalling 215mb. Loading this all on startup currently takes around 30s on a somewhat fast PC, more on a slower PC, so I would like to bring that down and have better loading times. So I'm wondering: what's the fastest way of loading large numbers of textures in bulk on startup?

My current way to force Ogre to really load the textures into video memory (instead of just declaring the resource and not loading it until it's actually used, which causes hickups while playing) is roughly like this:

Code: Select all

// Create resourceGroup
auto& rGM = Ogre::ResourceGroupManager::getSingleton();
rGM.createResourceGroup(groupName, true);
rGM.addResourceLocation(folder, "FileSystem", groupName);
rGM.initialiseResourceGroup(groupName);
rGM.loadResourceGroup(groupName);

// Force Ogre to fully load each texture by temporarily applying it to an actual material
auto& tempMaterial = static_cast<Ogre::MaterialPtr>(Ogre::MaterialManager::getSingleton().getByName(materialName))
auto textureUnitState = tempMaterial->getTechnique(0)->getPass(0)->getTextureUnitState(0);
std::vector<std::string> files = Tools::listDirectory(textureFolder, "");
for (const auto& fileName : files)
	textureUnitState->setTextureName(fileName);
My first hunch was to put them all in a ZIP file to avoid opening so many separate files on disk. However, to my surprise this made the loading time much longer. I'm guessing that's because Ogre needed to find each texture when used, instead of just going through the files in order and loading each texture?

Also, I tried running a profiler (on the non-ZIP version) and it turns out 42% of the time was spent in Ogre::PixelUtil::packColour. I had a look at the Ogre source and I think this means that Ogre is actually unpacking the DDS files instead of loading them in bulk. That wastes a lot of memory and a lot of loading time, how can I make Ogre actually use the DDS format internally? Is that maybe because my DDS files don't have mipmaps stored?

Based on experiences with Blightbound and Awesomenauts in our own engine I know such amounts can be loaded much faster than this, so I'm guessing I'm just not doing it the right way for Ogre. I'm guessing I need two things:
  • Make Ogre load all textures from an archive (compressed or not) in order, for maximum disk access speed
  • Make Ogre use DDS as DDS internally instead of unpacking it to uncompressed
So, what is the fastest way of loading this many textures in Ogre?
Last edited by Oogst on Fri May 07, 2021 3:25 pm, edited 2 times in total.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

can you post an excerpt of your ogre.log. Particularly the part where you load the texture:

Code: Select all

Texture 'BumpyMetal_dxt1.dds': Loading 1 faces(PF_DXT1,512x512x1) with 9 hardware generated mipmaps from Image. Internal format is PF_DXT1,512x512x1.
and the texture compression capabilities:

Code: Select all

 * Texture Compression: yes
   - DXT: yes
   - VTC: yes
   - PVRTC: no
   - ATC: no
   - ETC1: no
   - ETC2: yes
   - BC4/BC5: yes
   - BC6H/BC7: yes
   - ASTC: no
   - Mipmaps for compressed formats: yes
in short: you were put on the slow path here:
https://github.com/OGRECave/ogre/blob/3 ... c.cpp#L843

also, force loading a texture can be done simpler with:

Code: Select all

Ogre::TextureManager::getSingleton().load(...);
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Fastest way to load large number of textures

Post by Oogst »

These are the relevant bits from the log:

Code: Select all

RenderSystem Name: Direct3D11 Rendering Subsystem
GPU Vendor: nvidia
Device Name: NVIDIA GeForce GTX 970
Driver Version: 27.21.14.5671

Code: Select all

* Texture Compression: yes
  - DXT: yes
  - VTC: no
  - PVRTC: no
  - ATC: no
  - ETC1: no
  - ETC2: no
  - BC4/BC5: no
  - BC6H/BC7: yes
  - ASTC: no
  - Mipmaps for compressed formats: no
What does "Mipmaps for compressed formats: no" mean? I'd expect a Geforce 970 would be capable of doing DXT1 with mipmaps, right?

And here are two random textures (the rest is similar):

Code: Select all

Texture 'BodyOutlineSpline01_lightmap.dds': Loading 1 faces(PF_B8G8R8,512x512x1) with 5 generated mipmaps from Image. Internal format is PF_A8B8G8R8,512x512x1.
Texture 'Box01_lightmap.dds': Loading 1 faces(PF_B8G8R8,1024x1024x1) with 5 generated mipmaps from Image. Internal format is PF_A8B8G8R8,1024x1024x1.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5292
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Fastest way to load large number of textures

Post by dark_sylinc »

Oogst wrote: Tue May 04, 2021 6:43 pm And here are two random textures (the rest is similar):

Code: Select all

Texture 'BodyOutlineSpline01_lightmap.dds': Loading 1 faces(PF_B8G8R8,512x512x1) with 5 generated mipmaps from Image. Internal format is PF_A8B8G8R8,512x512x1.
Texture 'Box01_lightmap.dds': Loading 1 faces(PF_B8G8R8,1024x1024x1) with 5 generated mipmaps from Image. Internal format is PF_A8B8G8R8,1024x1024x1.
Generate the mipmaps offline. It will increase the size of the file by 33% though.
You're wasting time generating them on the fly.

texconv supports mipmap generation.
Oogst wrote: Tue May 04, 2021 6:43 pm What does "Mipmaps for compressed formats: no" mean? I'd expect a Geforce 970 would be capable of doing DXT1 with mipmaps, right?
Yeah that card supports it. This is probably an Ogre bug, but I don't maintain 1.x. Btw mipmaps for compressed formats is very old. I'd expect any GPU bought in the last 15 years to support it.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Fastest way to load large number of textures

Post by Oogst »

I've generated the mipmaps offline now and that indeed destroys the loading times: texture loading is around 6x faster now. :D

For some textures I actually want them to not have mipmaps, because they're a weird kind of texture atlas. Is there some way to tell Ogre that I don't want mipmaps on a specific texture? I know I can define it in the material, but then the texture will still be loaded with mipmaps, they just won't be used.

Also, is there some way to load all the textures from a single archive in one big swoop, to avoid the price of opening a lot of small files? The code I posted in my starting post is very slow when I use a Zip instead of a FileSystem, so I'm guessing that should be done in a different way?
Last edited by Oogst on Tue May 04, 2021 7:08 pm, edited 1 time in total.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
loath
Platinum Sponsor
Platinum Sponsor
Posts: 290
Joined: Tue Jan 17, 2012 5:18 am
x 67

Re: Fastest way to load large number of textures

Post by loath »

i bet packing the textures into a single uncompressed file is the fastest. ex. like torchlight does with it's pak files. as you stream the files out of the pak (via a sequential read) you could hand off batches of files to background threads for processing.

i don't know if the dx11 render system allows loading textures on background threads but here is what i do on directx 9:
1. load the individual files off the disk into Ogre::Images on c++ worker threads and then
2. convert them to Ogre::Textures when the time is right on the main thread. (basically i trickle a few Ogre::Images into Ogre::Textures over several frames so there are no frame rate hiccups).
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

dark_sylinc wrote: Tue May 04, 2021 6:56 pm
Oogst wrote: Tue May 04, 2021 6:43 pm What does "Mipmaps for compressed formats: no" mean? I'd expect a Geforce 970 would be capable of doing DXT1 with mipmaps, right?
Yeah that card supports it. This is probably an Ogre bug, but I don't maintain 1.x. Btw mipmaps for compressed formats is very old. I'd expect any GPU bought in the last 15 years to support it.
yes, this is a bug indeed. I forgot to add the D3D RenderSystems when I introduced RSC_AUTOMIPMAP_COMPRESSED.
This was needed as neither GLES2 nor ES3 support mipmaps for compressed formats.

On the bright side, the fix is a on-liner:
https://github.com/OGRECave/ogre/pull/2004/files

edit:
actually, no! that is not a bug. Neither D3D11 does support compressed formats:
https://docs.microsoft.com/en-us/window ... neratemips

just dug through git history and I removed that RSC on purpose.
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

Oogst wrote: Tue May 04, 2021 7:06 pm For some textures I actually want them to not have mipmaps, because they're a weird kind of texture atlas. Is there some way to tell Ogre that I don't want mipmaps on a specific texture?
https://ogrecave.github.io/ogre/api/lat ... fcfc86d197
Oogst wrote: Tue May 04, 2021 7:06 pm Also, is there some way to load all the textures from a single archive in one big swoop, to avoid the price of opening a lot of small files? The code I posted in my starting post is very slow when I use a Zip instead of a FileSystem, so I'm guessing that should be done in a different way?
Maybe if you profile the Zip version, this would give us some insights. Basically Zip is the container you want - it should be fast for low compression values.
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

loath wrote: Tue May 04, 2021 7:08 pm i don't know if the dx11 render system allows loading textures on background threads but here is what i do on directx 9:
1. load the individual files off the disk into Ogre::Images on c++ worker threads and then
that should be equivalent to just calling Texture::prepare. Does that give you any issues?
loath
Platinum Sponsor
Platinum Sponsor
Posts: 290
Joined: Tue Jan 17, 2012 5:18 am
x 67

Re: Fastest way to load large number of textures

Post by loath »

good point, prepare looks like it would work fine.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Fastest way to load large number of textures

Post by Oogst »

paroj wrote: Tue May 04, 2021 8:06 pm [...]
actually, no! that is not a bug. Neither D3D11 does support compressed formats:
https://docs.microsoft.com/en-us/window ... neratemips
[...]
I think DX11 does support having mipmaps on compressed DXT textures? Don't you mean that DX11 doesn't support automatically generating mipmaps for compressed textures? In that case it might be better to rename the capability logging from "Mipmaps for compressed formats" to something like "Automatic mipmap generation for compressed formats" to avoid confusion.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Fastest way to load large number of textures

Post by Oogst »

I've experimented some more and now I'm getting different results: on three different computers, an uncompressed ZIP is always faster than reading the files separately from disk, just like paroj said it would be. I have no idea why I saw something different earlier. The speed increase varies a lot, but on an older laptop I tried it's as much as 2x faster.

Note that measuring startup times is tricky because Windows 10 likes to keep files in memory after use, so that a second startup is much faster. In my testing I saw one case of the second startup being 30x faster! So for better measuring I've made sure to reboot before every test, and then wait a while for all Windows stuff to have been initialised.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

Oogst wrote: Wed May 05, 2021 8:25 am
paroj wrote: Tue May 04, 2021 8:06 pm [...]
actually, no! that is not a bug. Neither D3D11 does support compressed formats:
https://docs.microsoft.com/en-us/window ... neratemips
[...]
I think DX11 does support having mipmaps on compressed DXT textures? Don't you mean that DX11 doesn't support automatically generating mipmaps for compressed textures? In that case it might be better to rename the capability logging from "Mipmaps for compressed formats" to something like "Automatic mipmap generation for compressed formats" to avoid confusion.
yes, I admit that the log wording is confusing. The RSC_AUTOMIPMAP_COMPRESSED reveals what this is about though.

I was also thinking about changing the behavior when encountering DXT w/o mips: Instead of unpacking to generate mips, we probably should just disable mipmapping.
If one cares about texture compression, one probably knows what he is doing. At last it would meet your expectations in the case here.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: [Solved] Fastest way to load large number of textures

Post by Oogst »

Hmm, how do I actually turn off using mipmaps in a material for a texture that does have mipmaps? I had expected this would do the trick:

Code: Select all

filtering linear linear none
But I still get mipmaps that way. I can truly get the top mipmap when I use this:

Code: Select all

mipmap_bias -100
But that seems like an inefficient workaround and not a real solution.

For reference, this is the complete material:

Code: Select all

material CGMine
{
	technique
	{
		pass
		{
			vertex_program_ref CGMainMaterialVP
			{
			}
			fragment_program_ref CGMineFP
			{
			}
			texture_unit colourTexture
			{
				texture Mines.png
				filtering anisotropic
				max_anisotropy 16
			}
			texture_unit lightmapTexture
			{
				texture Mine_atlas_lightmap.dds
				filtering linear linear none
				tex_address_mode clamp
			}
			texture_unit selfIlluminationTexture
			{
				texture Mine_selfIllumination_lightmap.dds
				filtering anisotropic
				max_anisotropy 16
				tex_address_mode clamp
			}
		}
	}
}
The "lightmapTexture" is the one that I want to disable mipmaps on.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5292
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [Solved] Fastest way to load large number of textures

Post by dark_sylinc »

Declare "texture Mine_atlas_lightmap.dds 0" (see manual, Attribute Descriptions section)

However note that if there are two materials using the same texture and they differ in their mipmap usage; the material to get loaded first (or was it parsed first?) wins.
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: [Solved] Fastest way to load large number of textures

Post by Oogst »

dark_sylinc wrote: Fri May 07, 2021 1:54 pm Declare "texture Mine_atlas_lightmap.dds 0" (see manual, Attribute Descriptions section) [...]
Setting that has a rather odd result. I had hoped it would just use mipmap level 0 at all times, but instead it seems to use some generic grey value for all other mipmap levels. If I get closer to the object, I can see it get the lightmap per triangle as it switches to mipmap level 0. If I zoom out, it's all grey.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

did you specify both

Code: Select all

				texture Mine_atlas_lightmap.dds 0
				filtering linear linear none
?
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Fastest way to load large number of textures

Post by Oogst »

Yes. I used the material definition above, with as the only change the added zero.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

seems mipmapping cannot be disabled on D3D11 like that, as we cannot actually map "linear linear none":
https://docs.microsoft.com/en-us/window ... d11_filter

edit:
on the other hand "texture Mine_atlas_lightmap.dds 0" should be sufficient to get no mipmapping.

maybe a bug in ogre. Can you check with https://renderdoc.org/ that there are indeed no mip levels allocated?
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1067
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
x 43
Contact:

Re: Fastest way to load large number of textures

Post by Oogst »

According to Nvidia Nsight the texture in question does have mipmaps. The original DDS file on disk also has mipmaps. For clarity, this is the complete material that I use right now:

Code: Select all

material CGMine
{
	technique
	{
		pass
		{
			vertex_program_ref CGMainMaterialVP
			{
			}
			fragment_program_ref CGMineFP
			{
			}
			texture_unit colourTexture
			{
				texture Mines.png
				filtering anisotropic
				max_anisotropy 16
			}
			texture_unit lightmapTexture
			{
				texture Mine_atlas_lightmap.dds 0
				filtering linear linear none
				tex_address_mode clamp
			}
			texture_unit selfIlluminationTexture
			{
				texture Grey.tga
				filtering anisotropic
				max_anisotropy 16
				tex_address_mode clamp
			}
		}
	}
}
Since the game isn't GPU heavy anyway, I guess I can just leave it at mipmap_bias -100, since that does achieve what I want. I'm assuming the price of that over simply only using mipmap 0 isn't that big of a difference, since the resulting texture read is exactly the same anyway.
My dev blog
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/X1/PS3/PS4)
Blightbound: coop online dungeon crawler (PC)
Swords & Soldiers: side-scrolling RTS (Switch/PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance
The Ageless Gate: cello album
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Fastest way to load large number of textures

Post by paroj »

ah.. yes.. your DDS have mips. Those override your manually specified number.

You can get rid of the DDS mips again, if you apply this patch:
https://github.com/OGRECave/ogre/commit ... 6dadc0a8be

edit:
and this is the patch for the issue at hand: https://github.com/OGRECave/ogre/pull/2010/files
Post Reply