Ogre heap and boost::serialization

[xavier]

Anyone using the Boost serialization library with Ogre?

I am not entirely sure how either

1) to make Ogre quit hijacking the "new" operator used in the Boost library code, or
2) make the Boost code use Ogre's "delete" as well as its "new"

The problem is an unrelenting DbgHeap assert when my deserialized class goes out of scope (program end, manual scoping, etc). I get the typical "you allocated this on one heap and are trying to dealloc it on another" ('another' being the "normal" CRT heap) assertion in std::locale's dtor, as created by boost::archive::binary_iarchive (which is pure template, no compiled component). This is linking against the static Boost serialization library, so I know it's not that; I have isolated it down to OgreMain, which as we know defines "operator new" and "operator delete" in the generic namespace (so I cannot even turn it off with namespace manipulation).

This has been driving me batty for days now. No combination of Boost library and MSVC (7.1) code generation settings works. Any ideas would help.

Thanks!

[sinbad]

Just make sure you include boost headers before Ogre in all cases, or surround the boost headers with

Code: Select all

#include "OgreNoMemoryMacros.h"
.. boost
#include "OgreMemoryMacros.h"

[xavier]

Thanks for the response, Sinbad, but I tried all permutations of that as well. I still get the following trace when the archive goes out of scope:

> cosD.exe!_CrtIsValidHeapPointer(const void * pUserData=0x00c00190) Line 1807 C
cosD.exe!_free_dbg(void * pUserData=0x00c00190, int nBlockUse=1) Line 1132 + 0x9 C
cosD.exe!free(void * pUserData=0x00c00190) Line 1025 + 0xb C
cosD.exe!std::_DebugHeapDelete<std::locale::facet>(std::locale::facet * _Ptr=0x00c00190) Line 63 + 0x9 C++
cosD.exe!std::locale::_Locimp::~_Locimp() Line 173 + 0x17 C++
cosD.exe!std::locale::_Locimp::`scalar deleting destructor'() + 0xf C++
cosD.exe!boost::archive::basic_binary_iprimitive<boost::archive::binary_iarchive,std::basic_istream<char,std::char_traits<char> > >::~basic_binary_iprimitive<boost::archive::binary_iarchive,std::basic_istream<char,std::char_traits<char> > >() + 0x67 C++
cosD.exe!boost::archive::binary_iarchive_impl<boost::archive::binary_iarchive>::~binary_iarchive_impl<boost::archive::binary_iarchive>() + 0x77 C++
cosD.exe!boost::archive::binary_iarchive::~binary_iarchive() + 0x2b C++
cosD.exe!WinMain(HINSTANCE__ * hInst=0x00400000, HINSTANCE__ * __formal=0x00000000, char * strCmdLine=0x00141f03, HINSTANCE__ * __formal=0x00000000) Line 38 + 0xf C++
cosD.exe!WinMainCRTStartup() Line 251 + 0x30 C



I'd be perfectly happy to use the Ogre delete operator as well as the new operator, I just am not sure how to make that happen.

I was using boost::serialization as a handy and relatively "standard" way of storing and loading a mission file, but if I have to I will just implement the serialization manually (removes the need for RTTI to be enabled as well).

I should point out that this only occurs when the Ogre lib is linked in; test apps not using Ogre work flawlessly.

Thanks!
Greg

[Haibokusha]

I'm not sure if this has been fixed in 1.0RC, but I've always had problems with OGRE's memory allocator. The first thing I do when I download a new release is comment out most of the OgreMemory* files, as even requesting via preprocessor macros that the allocation functions not be overridden doesn't (didn't?) work. If I don't take this step, my programs start behaving very, very strangely (eg sometimes allocators return invalid locations, other times they just return zero without raising any other error conditions, etc).

[xavier]

In this case, I simply ended up implementing a method of serialization that does not rely on Boost at all.

I would advise against commenting out the Ogre memory manager unless you have your own somewhere else; modern CRT memory mgrs are much better than they were in years past, but for allocating/deallocating small things, in high volume, you want a custom memory manager for your app.

[sinbad]

The memory manager is there to catch leaks, it's not for performance. It doesn't get used in release builds. I've had issues with other libs that try to implement their own memory managers (e.g. Xerces) but have found they're always avoidable with those techniques above.

[Attis SH]

I had to disable the Debug memory manager too because I made a memory manager optimised for small buffer allocation for my String class and other small things like ::std::list nodes. I wanted to use this because it gave a noticeable performance increase when allocating/deallocating objects very often, like when dealing with strings for example.

It would be nice to have a memory manager that doesn't use macros that is only used inside Ogre. I know there is an approach for Visual Studio to implement this, and I will see if I can make it work some time later. The basic idea is to save the return address pointer (function caller pointer) and to use dbghelp functions to show function names and source lines when needed. It's much better in my opinion than the macro based approach. I'm also sure something similar could be implemented with gcc and libgdb as well.

[AssiDragon]

Errr if you make the OGRE memory manager only work in OGRE, do that as an option only. It makes a good job for me, catching the memory leaks of my code as well.

[Attis SH]

Errr if you make the OGRE memory manager only work in OGRE, do that as an option only. It makes a good job for me, catching the memory leaks of my code as well.

Absolutely, I didn't meant to remove something so useful. I'd be happy to use the memory debugger too, but it's current macro based approach make it useless for me, as I want to define new/delete operators for the base class of my memory managed objects.

I haven't looked into the possibilities yet but if it could be implemented by redefining the normal new/delete operators inside the Ogre namespace all Ogre classes could use it. Then if you want to use them you could define the same operators inside your own namespace or even at global level.

But as I said I haven't put together the pieces yet, I'm just speculating how it could work...

[Haibokusha]

I've written an optimized allocator for various little things as well, but I don't override new/delete; instead, I have a sort of factory object that can produce and free its products via member functions. That way, I don't run into much nastiness with unusual things happening to the aforementioned operators.

[Attis SH]

I've written an optimized allocator for various little things as well, but I don't override new/delete; instead, I have a sort of factory object that can produce and free its products via member functions. That way, I don't run into much nastiness with unusual things happening to the aforementioned operators.

Sorry but I didn't remember exactly what was my problem with memory macros. My memory manager doesn't override any new/delete itself either, it has a simple void* alloc(size_t) and a void dealloc(void*) function. The problems appeared at places where I tried to use it.

My problem is related to my string implementation where I need to use placement new to write things down properly. Basically I allocate a single refcounted memory buffer with both data and string character array in a single allocation, and use placement new on this buffer. The memory macros made this impossible.

Also, to simplify usage of the optimized memory manager I would make a simple base class called FastAllocated which overriden new/delete operators, and subclass from it where needed. Using factories one would need more code and/or macros to have similar functionality.

Factories (recyclers) use a different approach than mine. I don't like to use them for recycling because objects of the same size can't usually share the same recycler instance, and it's more problematic to use them for a lot of classes. There are places where their use them for extra performance, like for Billboards in Ogre.

[rincewind]

Hi,

just stumbled across the same problem with boost::serialization. Did anybody ever find a solutions (short of disabling the memory manager).?
I've been clutterning my code already with "OgreNoMemoryMacros.h" but it doesn't really help.

I really like to have the memory manager to detect leaks.

Rincewind

[BenO]

bump

i have the same pb . anyone got a solution ?

[jacmoe]

i have the same pb . anyone got a solution ?

A problem with boost::serialization ?
Will try and get it to work. I am quite interested in using that library myself.

[BenO]

yeah :) boost::serialization.

i disabled the ogre memory manager : and it works fine.

But i would be happy to debug my application with serialization
and leak detection :o


Ps: if u plan on using it, i suggest u download the CVS version :
The last release is quite old.

[:wumpus:]

I generally disable the Ogre memory manager as well nowadays, it interferes with almost every other library (Sh, Boost, ...) due to the wacky free/alloc macro definitions.
I'm sure there is a better way to do memory debugging than having to compile everything with some macros.

[BenO]

good point

[Game_Ender]

I'm sure there is a better way to do memory debugging than having to compile everything with some macros.

It seems like I have been pimping this program a lot lately, but if you develope on linux you could just run your program through valgrind. It has does has done a very good job of catching memory leaks for me in the past. It catches things like using uninitialized values and other little memory errors that cause undefined behavior. The only problem is it might make Ogre so slow that you wouldn't be able to test.

[getli]

Bump. Ran into the same problem with boost serialization

Code: Select all

#include "OgreNoMemoryMacros.h"
.. boost
#include "OgreMemoryMacros.h" 

doesn't work. Including before the Ogre includes doesn't either. Has anyone solved this? (Disabling the memory manager works, but is not desirable in debug builds for me).

[xavier]

Never did. I ended up "rolling-our-own" serialization solution for the classes in question.

[getli]

That's a shame. Boost serialisation is easy to use and well engineered. I don't want to reinvent the wheel, but also don't want to loose the leak checking. Could anyone of the Ogre team look into that? Surely there must be a way to make the memory manager more compatible with other libraries. One radical way I've used in the past would be replacing new by something like OGRE_NEW, so only allocations done with OGRE_NEW/OGRE_MALLOC would be tracked. This way you can be sure that external libraries work fine. They don't get tracked but IMO that's a small tradeoff. I know this change would be a big one, but might be the cleanest solution to solve this problem (I am sure boost serialisation is not the only problem case as overloading/defining global new and delete is normally not encouraged).

[Praetor]

Yeah, I'm planning on using serialization as well, but I haven't yet run into problems. To be fair the serialization happens at a level above Ogre (in a wrapping engine) so it could help. I'll probably grab the source release of Ogre soon and compile it without the memory manager on to avoid any problems. I'll be using an external app (like valgrind) to catch memory leaks anyway.

I don't think there's any way around it without a lot of work. You'd have to override the new/delete operators for EVERY class in Ogre instead of once for the global ones to make the memory manager Ogre-specific, and even then it might interfere with things.

[getli]

I wasn't talking about overloading the member new/delete operators, but more about doing smth like this:

Code: Select all

//------------------------------------------------------------------------------
void* operator new(size_t size, const char* file, unsigned int line)
//------------------------------------------------------------------------------
{
    return _malloc_dbg(size, _NORMAL_BLOCK, file, line);
}

//------------------------------------------------------------------------------
void operator delete(void* pointer, const char*, unsigned int)
//------------------------------------------------------------------------------
{
    _free_dbg(pointer, _NORMAL_BLOCK);
}

#ifndef NDEBUG
#   define OGRE_NEW new(__FILE__, __LINE__)
#   define OGRE_DELETE delete
#   define OGRE_DELETE_A delete[]
#else // !NDEBUG
#   define OGRE_NEW new
#   define OGRE_DELETE delete
#   define OGRE_DELETE_A delete[]
#endif // !NDEBUG

Then you'd simply do a replace in files with new -> OGRE_NEW, delete -> OGRE_DELETE. As a client you'd use new for untracked and OGRE_NEW for tracked memory. Nice and clean and does not interfere with external libraries at all. If you want to track STL or smth similar you simply provide custom allocators that use OGRE_NEW instead of new

[OvermindDL1]

I've been having a lot of issues (and I mean a metric ton) with OGRE's memory manager and other libraries recently. I've just started seperating out my engine code from graphic code in full, fixes that right up, lot more coding though.

[grizzley90]

Has this problem ever been solved? Or is the best way to simply just remove the ogre memory manager?