[2.2] Memory leaking on render

Problems building or running the engine, queries about how to use features etc.
Post Reply
ori30ffs
Gnoblar
Posts: 11
Joined: Tue Sep 18, 2018 6:38 am

[2.2] Memory leaking on render

Post by ori30ffs »

Ogre Version: 2.2.5
Operating System: Windows
Render System: Directx11

Hello guys,
I have a problem with resource releasing. My project contains several model and each model should be loaded consistently after unload previous.
Model has v1 mesh format and I convert it to v2 with loadV1Model function. (really no matter which format I use, for v1 I have the same problem).
In the end of rendering, my application has size more than 3GB, but I expect ~700MB.

As I understand Ogre::TextureGpuManager still have data. How to release it?

Code: Select all


22:12:41: ===============================Start dump of TextureGpuManager===============================
22:12:41: == Dumping Pools ==
22:12:41: ||Width|Height|Format|Mipmaps|Size in bytes|Num. active textures|Total texture capacity|Pool ID|Texture Names
22:12:41: ||64|64|PFG_RGBA16_FLOAT|1|98304|3|3|992044|ltcMatrix0.dds|ltcMatrix1.dds|brtfLutDfg.dds
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_0.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_1.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_2.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_3.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_4.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_5.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_6.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_0_7.jpg
22:12:41: ||8192|8192|PFG_RGBA8_UNORM_SRGB|1|268435456|1|1|0|texture_BAB_1_0.jpg
22:12:41: ||4096|4096|PFG_RGBA8_UNORM_SRGB|1|134217728|2|2|0|texture_BAB_1_1.jpg|texture_BAB_1_2.jpg
22:12:41: ||4096|4096|PFG_RGBA8_UNORM_SRGB|1|134217728|2|2|0|texture_BAB_2_0.jpg|texture_BAB_2_1.jpg
22:12:41: ||2048|2048|PFG_RGBA8_UNORM_SRGB|1|33554432|1|2|0|texture_BAB_3_0.jpg
22:12:41: ||1024|1024|PFG_RGBA8_UNORM_SRGB|1|16777216|1|4|0|texture_BAB_3_1.jpg
22:12:41: |Alias|Resource Name|Width|Height|Depth|Num Slices|Format|Mipmaps|MSAA|Size in bytes|RTT|UAV|Manual|MSAA Explicit|Reinterpretable|AutomaticBatched|Residency
22:12:41: |texture_BAB_0_1.jpg|texture_BAB_0_1.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_2_1.jpg|texture_BAB_2_1.jpg|4096|4096|1|1|PFG_RGBA8_UNORM_SRGB|1|1|67108864|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_0_7.jpg|texture_BAB_0_7.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_0_3.jpg|texture_BAB_0_3.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_1_2.jpg|texture_BAB_1_2.jpg|4096|4096|1|1|PFG_RGBA8_UNORM_SRGB|1|1|67108864|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_0_4.jpg|texture_BAB_0_4.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_0_5.jpg|texture_BAB_0_5.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_1_1.jpg|texture_BAB_1_1.jpg|4096|4096|1|1|PFG_RGBA8_UNORM_SRGB|1|1|67108864|0|0|0|0|0|1|Resident
22:12:41: |ltcMatrix1.dds|ltcMatrix1.dds|64|64|1|1|PFG_RGBA16_FLOAT|1|1|32768|0|0|0|0|0|1|Resident
22:12:41: |brtfLutDfg.dds|brtfLutDfg.dds|64|64|1|1|PFG_RGBA16_FLOAT|1|1|32768|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_0_6.jpg|texture_BAB_0_6.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_1_0.jpg|texture_BAB_1_0.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_3_1.jpg|texture_BAB_3_1.jpg|1024|1024|1|1|PFG_RGBA8_UNORM_SRGB|1|1|4194304|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_0_2.jpg|texture_BAB_0_2.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_3_0.jpg|texture_BAB_3_0.jpg|2048|2048|1|1|PFG_RGBA8_UNORM_SRGB|1|1|16777216|0|0|0|0|0|1|Resident
22:12:41: |ltcMatrix0.dds|ltcMatrix0.dds|64|64|1|1|PFG_RGBA16_FLOAT|1|1|32768|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_2_0.jpg|texture_BAB_2_0.jpg|4096|4096|1|1|PFG_RGBA8_UNORM_SRGB|1|1|67108864|0|0|0|0|0|1|Resident
22:12:41: |texture_BAB_0_0.jpg|texture_BAB_0_0.jpg|8192|8192|1|1|PFG_RGBA8_UNORM_SRGB|1|1|268435456|0|0|0|0|0|1|Resident
22:12:41: 
|MBs in pools (excluding waste):|2580.09
|MBs in pools (including waste):|2608.09
|MBs outside of pools:|0.00
|Total MBs (excl. waste):|2580.09
|Total MBs (incl. waste):|2608.09
22:12:41: Available Staging Textures	|1|128 MB		 |In use:	|176 MB
22:12:41: =============================End dump of TextureGpuManager=============================

Code: Select all

Ogre::MeshPtr loadV1Model(std::string name, std::string group)
{

    Ogre::v1::MeshPtr meshV1 = Ogre::v1::MeshManager::getSingleton().load(
        name, group,
        Ogre::v1::HardwareBuffer::HBU_DYNAMIC, Ogre::v1::HardwareBuffer::HBU_DYNAMIC);

    //Ogre::MeshPtr meshV2 = Ogre::MeshManager::getSingleton().createManual(name, group);
    // meshV2->importV1(meshV1.get(), false, false, true);
    bool halfPosition = false;
    bool halfUVs = false;
    bool useQTangents = false;
    const Ogre::String meshNameV2 = (name +Ogre::String(" Imported"));
    Ogre::MeshPtr meshV2 = Ogre::MeshManager::getSingleton().createByImportingV1(
        meshNameV2, group, meshV1.get(),
        halfPosition, halfUVs, useQTangents);
    meshV2->setVertexBufferPolicy(Ogre::BT_DEFAULT);
    meshV2->setIndexBufferPolicy(Ogre::BT_DEFAULT);
    meshV2->load();
    meshV1->unload();
    Ogre::v1::MeshManager::getSingleton().remove(meshV1);
    return meshV2;
}

int draw(std::string path_to_models, GraphicsSystem* mGraphicSystem)
{
    std::vector < std::tuple < std::string, std::string, double>> meta;
    int resource_count = 0;
    int index = 0;

    std::cout << "PROCESS: render" << std::endl;
    auto file_list = FindAllModelInDir(path_to_models);

    fs::path dir(path_to_models);
    for (auto f : file_list)
    {
        std::string filename = f.generic_string();
        fs::path file(filename);
        
        fs::path name = file.stem();
        fs::path full_path_dest = dir / name;

        std::string model_name = name.string();
        std::string model_path = full_path_dest.string();

        Ogre::ResourceGroupManager::getSingleton().addResourceLocation(model_path, "FileSystem", model_name, true, false);
        Ogre::ResourceGroupManager::getSingleton().initialiseResourceGroup(model_name, false);
        Ogre::MeshPtr mesh = loadV1Model(model_name + ".mesh", model_name);
        //Ogre::v1::MeshPtr mesh = Ogre::v1::MeshManager::getSingleton().load(
        //    model_name + ".mesh", model_name,
        //    Ogre::v1::HardwareBuffer::HBU_DYNAMIC, Ogre::v1::HardwareBuffer::HBU_DYNAMIC);
        
        //Ogre::v1::Entity* item = mGraphicSystem->getSceneManager()->createEntity(mesh, Ogre::SCENE_STATIC);
        Ogre::Item* item = mGraphicSystem->getSceneManager()->createItem(mesh, Ogre::SCENE_STATIC);
        auto ogreNode = mGraphicSystem->getSceneManager()->getRootSceneNode(Ogre::SCENE_STATIC)->createChildSceneNode(Ogre::SCENE_STATIC);
        ogreNode->attachObject(item);
       
        Ogre::TextureGpuManager* textureMgr = mGraphicSystem->getRoot()->getRenderSystem()->getTextureGpuManager();
        textureMgr->waitForStreamingCompletion();
        mGraphicSystem->getRoot()->renderOneFrame();

        mGraphicSystem->getSceneManager()->destroyAllItems();
        Ogre::MeshManager::getSingleton().remove(mesh->getHandle());
        Ogre::ResourceGroupManager::getSingleton().removeResourceLocation(model_path, model_name);
        Ogre::ResourceGroupManager::getSingleton().destroyResourceGroup(model_name);

    }
    mGraphicSystem->getRoot()->renderOneFrame();

    Ogre::RenderSystem* renderSystem = mGraphicSystem->getRoot()->getRenderSystem();

    Ogre::TextureGpuManager* textureGpuManager = renderSystem->getTextureGpuManager();
    Ogre::LogManager* lm = new  Ogre::LogManager();
    auto log = lm->createLog("log.log", true, false, false);
    textureGpuManager->dumpMemoryUsage(log);

    return 0;
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.2] Memory leaking on render

Post by dark_sylinc »

Hi!

Destroying the Item/Entity and mesh won't destroy the materials and textures loaded with it.

You'll need to iterate the submeshes, find their materials and unload the textures used by those materials:

Code: Select all

Ogre::HlmsManager *hlmsManager = ...;
Ogre::Mesh *mesh = ...;
std::set<Ogre::TextureGpu *> usedTextures;

for( size_t i = 0u; i < mesh->getNumSubMeshes(); ++i )
{
    Ogre::SubMesh *subMesh = mesh->getSubMesh( i );
    Ogre::HlmsDatablock *datablock =
        hlmsManager->getDatablockNoDefault( subMesh->getMaterialName() );

    if( datablock->mType == Ogre::HLMS_PBS )
    {
        assert( dynamic_cast<Ogre::HlmsPbsDatablock *>( datablock ) );
        Ogre::HlmsPbsDatablock *pbsDatablock =
            static_cast<Ogre::HlmsPbsDatablock *>( datablock );
        for( size_t j = 0u; j < Ogre::NUM_PBSM_TEXTURE_TYPES; ++j )
        {
            Ogre::TextureGpu *tex = pbsDatablock->getTexture( j );
            if( tex )
                usedTextures.insert( tex );
        }
    }
    else if( datablock->mType == Ogre::HLMS_UNLIT )
    {
        assert( dynamic_cast<Ogre::HlmsUnlitDatablock *>( datablock ) );
        Ogre::HlmsUnlitDatablock *unlitDatablock =
            static_cast<Ogre::HlmsUnlitDatablock *>( datablock );
        for( size_t j = 0u; j < Ogre::NUM_UNLIT_TEXTURE_TYPES; ++j )
        {
            Ogre::TextureGpu *tex = unlitDatablock->getTexture( j );
            if( tex )
                usedTextures.insert( tex );
        }
    }
}

for( auto &tex : usedTextures )
    tex->scheduleTransitionTo( Ogre::GpuResidency::OnStorage );
Cheers
Matias
ori30ffs
Gnoblar
Posts: 11
Joined: Tue Sep 18, 2018 6:38 am

Re: [2.2] Memory leaking on render

Post by ori30ffs »

Thank you!
In my case I also need to remove datablocks because my materials has the same naming for all models.

Code: Select all

Ogre::Root* root = mGraphicSystem->getRoot();
Ogre::HlmsManager* hlmsManager = root->getHlmsManager();
auto materialNames = destroyObject(mesh.get(), hlmsManager);
mGraphicSystem->getSceneManager()->destroyAllItems();
Ogre::MeshManager::getSingleton().remove(mesh->getHandle());
Ogre::ResourceGroupManager::getSingleton().removeResourceLocation(np_el.second, np_el.first);
Ogre::ResourceGroupManager::getSingleton().destroyResourceGroup(np_el.first);

for (auto elem : materialNames)
{
	Ogre::Hlms* hlms = hlmsManager->getHlms(static_cast<Ogre::HlmsTypes>(elem.second));
	hlms->destroyDatablock(elem.first);
}
Anyway, I have additional question. Is it possible to wait until all data will be unloaded? As I understood, unloading is async process, but in my application I no need to call render all time and just enough pause graphic core.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [2.2] Memory leaking on render

Post by dark_sylinc »

Your calls to waitForStreamingCompletion force it to be synchronous, both loading and unloading.

Btw only loading is async, unloading is sync. However if the texture is still loading in the background, then we must defer the destruction until it's done loading.

Since you're calling waitForStreamingCompletion to wait for it to finish loading, unloading should happen immediately.

Cheers
Matias
Post Reply