Problems loading textures and .dds file formats

[SolarPortal]

Hi,
I am experiencing crashes when loading some textures. They are mostly when loading from a .dds file for normal and spec
but i also get crashes when video memory runs out also limiting the amount textures, materials we can view in our engine using the asset manager before CTD.

Note: I would like to say that all these assets were loadable on the old DX9.

Here is my setup code:

Code: Select all

	// Create the item variant
	mesh.importV1Mesh("barbarian_high.mesh");

	// Load the HlmsJson file
	Ogre::HlmsManager* hlmsMgr = Ogre::Root::getSingleton().getHlmsManager();
	hlmsMgr->loadMaterials("barbarian_high_armor_mat.json", Ogre::ResourceGroupManager::AUTODETECT_RESOURCE_GROUP_NAME);
	hlmsMgr->loadMaterials("barbarian_high_eye_mat.json", Ogre::ResourceGroupManager::AUTODETECT_RESOURCE_GROUP_NAME);
	hlmsMgr->loadMaterials("barbarian_high_skin_mat.json", Ogre::ResourceGroupManager::AUTODETECT_RESOURCE_GROUP_NAME);

	node = rootNode->createChildSceneNode();
	Ogre::Item* item = scene.sceneManager->createItem("barbarian_high.mesh");
	node->attachObject(item);
	node->setPosition(-1, 1, 0);

@dark_sylinc, i have sent the mesh folder for you to reproduce
Here is the log section:

Code: Select all

Mesh: Loading barbarian_high.mesh.
Skeleton: Loading barbarian_high.skeleton
WARNING: barbarian_high.mesh is an older format ([MeshSerializer_v1.8]); you should upgrade it as soon as possible using the OgreMeshTool tool.
Texture: loading armor_d.dds as armor_d.dds
WARNING: Could not generate mipmaps for armor_d.dds. This can negatively impact performance as the HlmsTextureManager will create more texture arrays than necessary, and the lower mips won't be available. Lack of mipmaps also contribute to aliasing. If this is a compressed DDS/PVR file, bake the mipmaps offline.
Texture: loading barbarian_d.dds as barbarian_d.dds
WARNING: Could not generate mipmaps for barbarian_d.dds. This can negatively impact performance as the HlmsTextureManager will create more texture arrays than necessary, and the lower mips won't be available. Lack of mipmaps also contribute to aliasing. If this is a compressed DDS/PVR file, bake the mipmaps offline.
Texture: loading barbarian_s.dds as barbarian_s.dds
WARNING: Could not generate mipmaps for barbarian_s.dds. This can negatively impact performance as the HlmsTextureManager will create more texture arrays than necessary, and the lower mips won't be available. Lack of mipmaps also contribute to aliasing. If this is a compressed DDS/PVR file, bake the mipmaps offline.
OGRE EXCEPTION(9:UnimplementedException): This method can not be used to compress or decompress images in PixelUtil::bulkPixelConversion at E:\00_Skyline Integrations\Ogre3D SDK\Source\v2-1\OgreMain\src\OgrePixelFormat.cpp (line 958)

From where i know its crashing is in bulkPixelConversion.
Thanks again

[GlowingPotato]

What are you using to generate dds files ? I had some problem with some newer dds headers in the past.

[xrgo]

I remember having that exception when trying to pack a texture with an incompatible compression format, for example (not sure if is a valid example) dxt5 in the monochrome pack. It's not that the texture is bad/broken (since it worked fine in older Ogre)... but the HlmsTextureManager tries to pack that texture in an array with other textures, so it finds an array that has similar format and tries to pack it, and for that sometimes needs to convert it (bulkPixelConversion).

but i also get crashes when video memory runs out

that's strange, I only get super bad performance, never crash.
I also needed lots of video memory, I ended up using the same texture format for everything! (dxt1) and packed rough, metal and others in the same texture http://www.ogre3d.org/forums/viewtopic.php?f=25&t=84156. Also using PF_D32_FLOAT shadows reduced memory usage a lot http://www.ogre3d.org/forums/viewtopic.php?f=2&t=85281.

[GlowingPotato]

Same here with GPU memory. No crash but super bad performance. we are packing metallic and roughness together using DXT1 same for color, DXT5 for color+alpha, and BC5 for normal maps.
Our project is open world, so yeah, we are suffering with the memory pool. Texture streaming would make things a lot better

For reference, this is our GPU memory use when using textures at maximum quality. (still need to figure it out why our gpu use looks like a roller coaster.)

[al2950]

This is an area that I am currently having issues with. The problem is the HLMSTextureManager tries to pack texture types into a single texture array, it can convert texture types on the fly based on the preferred pixel format. Here are the defaults

Code: Select all

        mDefaultTextureParameters[TEXTURE_TYPE_DIFFUSE].hwGammaCorrection   = true;
        mDefaultTextureParameters[TEXTURE_TYPE_MONOCHROME].pixelFormat      = PF_L8;
        mDefaultTextureParameters[TEXTURE_TYPE_NORMALS].pixelFormat         = PF_BC5_SNORM;
        mDefaultTextureParameters[TEXTURE_TYPE_NORMALS].isNormalMap         = true;
        mDefaultTextureParameters[TEXTURE_TYPE_DETAIL].hwGammaCorrection    = true;
        mDefaultTextureParameters[TEXTURE_TYPE_DETAIL_NORMAL_MAP].pixelFormat=PF_BC5_SNORM;
        mDefaultTextureParameters[TEXTURE_TYPE_DETAIL_NORMAL_MAP].isNormalMap = true;
        mDefaultTextureParameters[TEXTURE_TYPE_ENV_MAP].hwGammaCorrection   = true;

As you can see the defaults explcicilty set MONOCHROME textures to PF_L8, the issues comes at the point when you tried to load a MONOCHROME texture which is compressed, Ogre will currently try and convert it to PF_L8 which it fails to do because it is compressed. There is some logic with normal maps to deal with this issue, however the logic needs to be made more generic so it does not cause crashes, but still be clear that ogre is unable to load textures efficiently (I mean performance not memory uses here)

I will get this fix done in the next couple of days as my team need it yesterday!

[SolarPortal]

@al2950, wow posting at the same time This fix would be really great to have and will add a lot of stability. Thanks

I have 1gb of dedicated video memory on a GTX560ti and windows uses around 200mb. When engine is loaded, around 400mb is used and then viewing material and meshes in our asset manager soon creeps up the memory usage until crashing usually in HlmsTextureManager::createOrRetrieve() or Datablock::setTexture().

What are you using to generate dds files ? I had some problem with some newer dds headers in the past.

Photoshop with either DXT1 for no alpha and DXT5 for alpha images.
Other users we have may use different pipelines so many texture formats are used overall either by badly designed game assets on game stores or saved in many formats so we need a high compatibility of load without crashing.

I remember having that exception when trying to pack a texture with an incompatible compression format, for example (not sure if is a valid example) dxt5 in the monochrome pack. It's not that the texture is bad/broken (since it worked fine in older Ogre)... but the HlmsTextureManager tries to pack that texture in an array with other textures, so it finds an array that has similar format and tries to pack it, and for that sometimes needs to convert it (bulkPixelConversion).

I actually had to reduce the texture packing into arrays to 1 per texture as it stopped as many crashes due to not using as much memory.
When the textures were packed into arrays and all loading into video memory when i only used 1 seemed crazy and extremely poor on video memory usage. It dropped like 300mb once i stopped using fully packed arrays.
Edit: With a large card, it would make sense

Also using PF_D32_FLOAT shadows reduced memory usage a lot viewtopic.php?f=2&t=85281.

Will have to check this out

[dark_sylinc]

Hi!

I took a look at the assets you sent me. Your normal maps are in BC1 format which we don't support for normal maps. Send them either in BC5S (signed BC5) or non-compressed (so we'll convert them to signed RG88).

I suggest you download texconv and use the command line to convert them to BC5S (do this with all your normal maps):

Code: Select all

texconv.exe -f BC5_SNORM armor_n.dds -o C:\OutputFolder

Since the normal map is already in a very lossy format (BC1) I suggest you use the original lossless map to convert to BC5S, otherwise the quality will be very poor.
Actually I suggest you bundle texconv.exe with your program, if the license allows it. That tool is a god-sent.

As for the out of memory problem, you may have noticed Ogre 2.1 is far more susceptible to this problem than previous versions. I suggest you read section 8.9 The Hlms Texture Manager of the manual, read it then post if you have questions.

Cheers

[al2950]

I took a look at the assets you sent me. Your normal maps are in BC1 format which we don't support for normal maps. Send them either in BC5S (signed BC5) or non-compressed (so we'll convert them to signed RG88).
s

On this note, is there any difference between SNORM & UNORM (apart from the obvious!). The reason I asked is because the current media tool I use always output UNORM DDS formats??

[GlowingPotato]

I couldn't fine a photoshop tool that export SNORM. So we are using the tool Matias told about. And I have no idea what signed and unsigned means.

[dark_sylinc]

I took a look at the assets you sent me. Your normal maps are in BC1 format which we don't support for normal maps. Send them either in BC5S (signed BC5) or non-compressed (so we'll convert them to signed RG88).
s

On this note, is there any difference between SNORM & UNORM (apart from the obvious!). The reason I asked is because the current media tool I use always output UNORM DDS formats??

Yes. UNORM maps the range [0; 255] to [0; 1]; while SNORM maps the range [-127; 127] to the range [-1; 1] (-128 and -127 both map to -1)
UNORM requires extra shader instructions to convert the range [0; -1] to [-1; 1]; basically it's (value * 2.0 - 1.0); while SNORM requires no extra instructions.

There is a new photoshop plugin from Intel. Perhaps it's worth a shot.

[GlowingPotato]

I took a look at the assets you sent me. Your normal maps are in BC1 format which we don't support for normal maps. Send them either in BC5S (signed BC5) or non-compressed (so we'll convert them to signed RG88).
s

On this note, is there any difference between SNORM & UNORM (apart from the obvious!). The reason I asked is because the current media tool I use always output UNORM DDS formats??

Yes. UNORM maps the range [0; 255] to [0; 1]; while SNORM maps the range [-127; 127] to the range [-1; 1] (-128 and -127 both map to -1)
UNORM requires extra shader instructions to convert the range [0; -1] to [-1; 1]; basically it's (value * 2.0 - 1.0); while SNORM requires no extra instructions.

There is a new photoshop plugin from Intel. Perhaps it's worth a shot.

I'am fallowing that intel plugin since early release because of modding fallout 4. And I'am afraid it only export UNORM.
But it does have a cool feature that is exporting linear instead of sRGB.

[SolarPortal]

As for the out of memory problem, you may have noticed Ogre 2.1 is far more susceptible to this problem than previous versions. I suggest you read section 8.9 The Hlms Texture Manager of the manual, read it then post if you have questions.

yes, i did read the section in the manual which is where i learned about the matTexturesPerArray and edited it. Setting the maxTextures to 1 stopped as many crashes happening due to lower dedicated vid mem, even though it may not be the best to do. The packing feature definately seems something that will benefit larger cards but become a hindrance on smaller cards.

I will check and use the texconv tool to convert these broken normals on the fly if the license allows it. I must admit that i am not up to scratch on these newer tex formats for the newer renderers.
Is there ways of stopping it crashing though and just show an error, ignore the file and let the other textures that are valid load rather than CTD. This would add stability at least.

We can adapt our workflow, but users who use our software already have setup libraries so any converting we can do to save them crash after crash the better lol

[al2950]

I took a look at the assets you sent me. Your normal maps are in BC1 format which we don't support for normal maps. Send them either in BC5S (signed BC5) or non-compressed (so we'll convert them to signed RG88).
s

On this note, is there any difference between SNORM & UNORM (apart from the obvious!). The reason I asked is because the current media tool I use always output UNORM DDS formats??

Yes. UNORM maps the range [0; 255] to [0; 1]; while SNORM maps the range [-127; 127] to the range [-1; 1] (-128 and -127 both map to -1)
UNORM requires extra shader instructions to convert the range [0; -1] to [-1; 1]; basically it's (value * 2.0 - 1.0); while SNORM requires no extra instructions.

There is a new photoshop plugin from Intel. Perhaps it's worth a shot.

Ok, that makes sense. Does our HLMS deal with that?

And as GlowingPotato said the intel plugin exports UNORM. On a side note it is a superb tool and I would highly recommend it over the old NVIDIA DDS photoshop plugin. (It is what I am using )

[al2950]

To answer my own post. No HLMS does not seem to deal with the UNORM SNORM issue for normal maps. However the HLMSTextureManager does, but NOT if you change the default pixel format to something other than PF_BC5_SNORM. So be careful!

[dark_sylinc]

To answer my own post. No HLMS does not seem to deal with the UNORM SNORM issue for normal maps. However the HLMSTextureManager does, but NOT if you change the default pixel format to something other than PF_BC5_SNORM. So be careful!

Actually what we try is to convert from UNORM to SNORM. The problem is that we don't support decompressing/compressing from and to compressed formats such as BC1, BC5, PRVTC, etc (and doing so on the fly would be slow & poor quality anyway). That's the actual problem.
For uncompressed formats (such as RGBA8888 png), we convert it to uncompressed SNORM RG88.

[SolarPortal]

just to note that .tga's can fail even on PBSM_DIFFUSE at times as well

[dark_sylinc]

just to note that .tga's can fail even on PBSM_DIFFUSE at times as well

What do you mean by that? Fail how?

[al2950]

Interesting to note, I had a look at GTA V and UE4 and neither use SNORM. GTA V use BC1 (which i was very surprised at!), unreal uses BC5_UNORM and converts in the shader.

Code: Select all

  13: sample_indexable(texture2d)(float,float,float,float) r3.xy, r3.xyxx, texture0.xyzw, sampler0
  14: add r4.xy, r3.xyxx, r3.xyxx
  15: mad r3.xy, r3.xyxx, l(2.000000, 2.000000, 0.000000, 0.000000), l(-1.000000, -1.000000, 0.000000, 0.000000)
  16: dp2 r0.w, r3.xyxx, r3.xyxx
  17: add r0.w, -r0.w, l(1.000000)
  18: max r0.w, r0.w, l(0)
  19: sqrt r4.z, r0.w
  20: add r4.xyz, r4.xyzx, l(-1.000000, -1.000000, 1.000000, 0.000000)

Iam intrigued as to why this would be, perhaps to be compatible with as much hardware as possible.....?

[SolarPortal]

Interesting about Gta5 and unreal.

@dark_sylinc, It again crashes when loading in from a JSON file. So I will let you know where it crashes today.

Edit: I have come in today and am not getting the same crash with the texture. I added a try catch around the hlmsMgr->loadMaterials(), so perhaps this caught an exception with duplicate datablock names.

Edit 2: Here is one of the out of memory errors:

Code: Select all

12:40:42: Texture: loading hk_mk23_01_D.tga as hk_mk23_01_D.tga
12:40:42: OGRE EXCEPTION(7:InternalErrorException): Texture Buffer: Out of memory in GL3PlusTextureBuffer::upload at E:\00_Skyline Integrations\Ogre3D SDK\Source\v2-1\RenderSystems\GL3Plus\src\OgreGL3PlusTextureBuffer.cpp (line 188)
12:40:42: OGRE EXCEPTION(7:InternalErrorException): Texture Buffer: Out of memory in GL3PlusTextureBuffer::upload at E:\00_Skyline Integrations\Ogre3D SDK\Source\v2-1\RenderSystems\GL3Plus\src\OgreGL3PlusTextureBuffer.cpp (line 188)
12:40:48: Texture: 4: Loading 1 faces(PF_A8B8G8R8,128x256x1) Internal format is PF_A8R8G8B8,128x256x1.
12:41:32: Texture: loading hk_mk23_01_N.tga as hk_mk23_01_N.tga

It looks as though the texture is loaded even after the exception has occurred.

I can catch the exception, but then it crashes straight after which i suspect is that normal map as .tga which has a same try catch as the diffuse.

If i load the assets in without the scene then the asset shows, but lets face it, no game is just 1 asset in a scene lol
I am sorry that i can't provide more information, based on i don't know what to provide as it just crashes at random points, not sure how a full level is going to load and work atm....
Do you want anything specific to work with?

As a thought and not sure how valid, but could you not add the ability to unload a texture that is not be used if a texture wants to be uploaded to the GPU and hasn't got enough memory?
Also just for a point of reference, my card can run Witcher3 at 20-25fps but never encounters an out of memory and shows the game in Med - High definition and the gpu is always maxed out and full vid mem usage.

[GlowingPotato]

As I said before, we don't have a crash when running out of memory, we have a turd machine with 1gb Vram, and what happens is the Dynamic memory kicks in. Performance goes down badly, but won't crash.

What you said about The Witcher, is pretty much their amazing asset streaming. Only textures and mips being rendered are loaded into the GPU. And textures are loaded from the last mipmap to the first one (no texture pop in). Same for LOD meshes.
And GTAV have that amazing world streaming. There is a bunch of material online about it.
A robust memory management inside ogre would be awesome.

[SolarPortal]

As I said before, we don't have a crash when running out of memory, we have a turd machine with 1gb Vram, and what happens is the Dynamic memory kicks in. Performance goes down badly, but won't crash.

Thanks, i do believe you are right and is what i would expect. I am checking to see what could still be causing the crash. A poor frame rate would be welcome any day over a crash.
Q) May i ask what your program is and what you have running etc...
Q) Also what card are you running?

I have noticed in OpenGL in Debug using Forward3D(unsure if it exists without) i get a bunch of warnings etc..
Here are the different errors:

Code: Select all

OpenGL:undefined behavior(medium) 131222: Program undefined behavior warning: Sampler object 2 has depth compare enabled but is bound to a non-depth texture, and used by a program that samples it with a shadow sampler. This is undefined behavior.

OpenGL:performance(medium) 131186: Buffer performance warning: Buffer object 15 (bound to GL_PIXEL_UNPACK_BUFFER_ARB, usage hint is GL_STATIC_DRAW) is being copied/moved from VIDEO memory to HOST memory.

The second one is what i expected which would not make it crash. Some also appear directly after texture loads.
There must be something larger in our source at play lol

Edit: Has the DefaultRaySceneQuery been updated yet, as when i use it in debug, i receive this from the OgreDefaultSceneQueries:

Code: Select all

                assert((!(objData.mVisibilityFlags[objData.mIndex] & VisibilityFlags::LAYER_VISIBILITY) ||
                        !(objData.mQueryFlags[objData.mIndex] & mQueryMask) ||
                        !objData.mOwner[j]->isCachedAabbOutOfDate()) &&
                        "Perform the queries after MovableObject::updateAllBounds has been called!");

RenderOneFrame is called on a timer update at the end of the update cycle.

[GlowingPotato]

Q) May i ask what your program is and what you have running etc...
Q) Also what card are you running?

We are developing a open world survival game. Running two ogre applications at the same time, a server and a client. Together, it uses 2.4GB of Vram when client is using high quality textures. Server is only rendering the GUI (MyGui) and uses ~340Mb Vram. We are still using V1 meshes, (entities) but looking forward for V2 and items. Performance its not great, but optimizations haven't been done yet. Client uses .DDS textures (Vram saver!!) that are generated from .TGA using a script.
The game is running in OpenGL (DX11 crashes like crazy for us, black textures... poor performance. But loads way faster then OpenGL). Our game Engine is custom made.

The turd machine uses a AMD 7850 1GB and 8GB ram. The others have a R9 280 3GB

[Zonder]

Server is only rendering the GUI (MyGui) and uses ~340Mb Vram.

Thats a lot of vram for the gui. Sure that's accurate?

[GlowingPotato]

Server is only rendering the GUI (MyGui) and uses ~340Mb Vram.

Thats a lot of vram for the gui. Sure that's accurate?

Right now its using 306MB.
Well, its only render the GUI, but needs to load meshes for collision testing. I can't confirm this because i'm not the programmer in here.

[SolarPortal]

Thanks, your project sounds interesting.

We are running a full v2 pipeline using items. DX11 and OpenGL3+ are running around the same, DX11 has lockups on loading of certain assets.
Perhaps this is an Nvidia issue with crashing. We try to use DDS where possible but as the topic started, normals crash if not in the correct format which we have yet to try the tool dark_sylinc has mentioned.

Without running a large scene, simple groundplane and directional light; In our asset browser, we display the resources in the folders by taking pictures of the assets so:
- If it is a mesh: it loads and applies the textures, takes a picture.
- If it is a material: it loads a sphere and applies the material and takes a snapshot.

Here is an example of a material folder that is being processed:


With no vid mem clearing and jumping from one asset to the other, it just eventually Crashes to Desktop.
We cannot even make it through 1 large folder of materials without it crashing with an out of memory or on a texture(which we know what causes the normals as mentioned before.).

I am beginning to suspect Nvidia cards. Can anyone test a Nvidia 1gb Vram card and fill the vid mem up using ogre? We don't own an AMD card to try with.

Edit: Running in Debug, i seem to be getting more images processed than release. Will get back with some more information.