Ogre heap and boost::serialization

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
User avatar
xavier
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 9481
Joined: Fri Feb 18, 2005 2:03 am
Location: Dublin, CA, US
x 22

Ogre heap and boost::serialization

Post by xavier »

Anyone using the Boost serialization library with Ogre?

I am not entirely sure how either

1) to make Ogre quit hijacking the "new" operator used in the Boost library code, or
2) make the Boost code use Ogre's "delete" as well as its "new"

The problem is an unrelenting DbgHeap assert when my deserialized class goes out of scope (program end, manual scoping, etc). I get the typical "you allocated this on one heap and are trying to dealloc it on another" ('another' being the "normal" CRT heap) assertion in std::locale's dtor, as created by boost::archive::binary_iarchive (which is pure template, no compiled component). This is linking against the static Boost serialization library, so I know it's not that; I have isolated it down to OgreMain, which as we know defines "operator new" and "operator delete" in the generic namespace (so I cannot even turn it off with namespace manipulation).

This has been driving me batty for days now. No combination of Boost library and MSVC (7.1) code generation settings works. Any ideas would help.

Thanks!
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

Just make sure you include boost headers before Ogre in all cases, or surround the boost headers with

Code: Select all

#include "OgreNoMemoryMacros.h"
.. boost
#include "OgreMemoryMacros.h"
User avatar
xavier
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 9481
Joined: Fri Feb 18, 2005 2:03 am
Location: Dublin, CA, US
x 22

Post by xavier »

Thanks for the response, Sinbad, but I tried all permutations of that as well. I still get the following trace when the archive goes out of scope:

> cosD.exe!_CrtIsValidHeapPointer(const void * pUserData=0x00c00190) Line 1807 C
cosD.exe!_free_dbg(void * pUserData=0x00c00190, int nBlockUse=1) Line 1132 + 0x9 C
cosD.exe!free(void * pUserData=0x00c00190) Line 1025 + 0xb C
cosD.exe!std::_DebugHeapDelete<std::locale::facet>(std::locale::facet * _Ptr=0x00c00190) Line 63 + 0x9 C++
cosD.exe!std::locale::_Locimp::~_Locimp() Line 173 + 0x17 C++
cosD.exe!std::locale::_Locimp::`scalar deleting destructor'() + 0xf C++
cosD.exe!boost::archive::basic_binary_iprimitive<boost::archive::binary_iarchive,std::basic_istream<char,std::char_traits<char> > >::~basic_binary_iprimitive<boost::archive::binary_iarchive,std::basic_istream<char,std::char_traits<char> > >() + 0x67 C++
cosD.exe!boost::archive::binary_iarchive_impl<boost::archive::binary_iarchive>::~binary_iarchive_impl<boost::archive::binary_iarchive>() + 0x77 C++
cosD.exe!boost::archive::binary_iarchive::~binary_iarchive() + 0x2b C++
cosD.exe!WinMain(HINSTANCE__ * hInst=0x00400000, HINSTANCE__ * __formal=0x00000000, char * strCmdLine=0x00141f03, HINSTANCE__ * __formal=0x00000000) Line 38 + 0xf C++
cosD.exe!WinMainCRTStartup() Line 251 + 0x30 C



I'd be perfectly happy to use the Ogre delete operator as well as the new operator, I just am not sure how to make that happen. :(

I was using boost::serialization as a handy and relatively "standard" way of storing and loading a mission file, but if I have to I will just implement the serialization manually (removes the need for RTTI to be enabled as well).

I should point out that this only occurs when the Ogre lib is linked in; test apps not using Ogre work flawlessly.

Thanks!
Greg
User avatar
Haibokusha
Kobold
Posts: 33
Joined: Mon Aug 09, 2004 4:45 am

Post by Haibokusha »

I'm not sure if this has been fixed in 1.0RC, but I've always had problems with OGRE's memory allocator. The first thing I do when I download a new release is comment out most of the OgreMemory* files, as even requesting via preprocessor macros that the allocation functions not be overridden doesn't (didn't?) work. If I don't take this step, my programs start behaving very, very strangely (eg sometimes allocators return invalid locations, other times they just return zero without raising any other error conditions, etc).
User avatar
xavier
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 9481
Joined: Fri Feb 18, 2005 2:03 am
Location: Dublin, CA, US
x 22

Post by xavier »

In this case, I simply ended up implementing a method of serialization that does not rely on Boost at all.

I would advise against commenting out the Ogre memory manager unless you have your own somewhere else; modern CRT memory mgrs are much better than they were in years past, but for allocating/deallocating small things, in high volume, you want a custom memory manager for your app.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Post by sinbad »

The memory manager is there to catch leaks, it's not for performance. It doesn't get used in release builds. I've had issues with other libs that try to implement their own memory managers (e.g. Xerces) but have found they're always avoidable with those techniques above.
User avatar
Attis SH
Kobold
Posts: 28
Joined: Sun May 18, 2003 9:42 am
Location: Budapest, Hungary

Post by Attis SH »

I had to disable the Debug memory manager too because I made a memory manager optimised for small buffer allocation for my String class and other small things like ::std::list nodes. I wanted to use this because it gave a noticeable performance increase when allocating/deallocating objects very often, like when dealing with strings for example.

It would be nice to have a memory manager that doesn't use macros that is only used inside Ogre. I know there is an approach for Visual Studio to implement this, and I will see if I can make it work some time later. The basic idea is to save the return address pointer (function caller pointer) and to use dbghelp functions to show function names and source lines when needed. It's much better in my opinion than the macro based approach. I'm also sure something similar could be implemented with gcc and libgdb as well.
AssiDragon
Greenskin
Posts: 145
Joined: Wed Apr 28, 2004 12:10 pm
Location: Hungary
Contact:

Post by AssiDragon »

Errr if you make the OGRE memory manager only work in OGRE, do that as an option only. It makes a good job for me, catching the memory leaks of my code as well.
Hope is the first step on the road to disappointment.
User avatar
Attis SH
Kobold
Posts: 28
Joined: Sun May 18, 2003 9:42 am
Location: Budapest, Hungary

Post by Attis SH »

AssiDragon wrote:Errr if you make the OGRE memory manager only work in OGRE, do that as an option only. It makes a good job for me, catching the memory leaks of my code as well.
Absolutely, I didn't meant to remove something so useful. I'd be happy to use the memory debugger too, but it's current macro based approach make it useless for me, as I want to define new/delete operators for the base class of my memory managed objects.

I haven't looked into the possibilities yet but if it could be implemented by redefining the normal new/delete operators inside the Ogre namespace all Ogre classes could use it. Then if you want to use them you could define the same operators inside your own namespace or even at global level.

But as I said I haven't put together the pieces yet, I'm just speculating how it could work...
User avatar
Haibokusha
Kobold
Posts: 33
Joined: Mon Aug 09, 2004 4:45 am

Post by Haibokusha »

I've written an optimized allocator for various little things as well, but I don't override new/delete; instead, I have a sort of factory object that can produce and free its products via member functions. That way, I don't run into much nastiness with unusual things happening to the aforementioned operators.
User avatar
Attis SH
Kobold
Posts: 28
Joined: Sun May 18, 2003 9:42 am
Location: Budapest, Hungary

Post by Attis SH »

Haibokusha wrote:I've written an optimized allocator for various little things as well, but I don't override new/delete; instead, I have a sort of factory object that can produce and free its products via member functions. That way, I don't run into much nastiness with unusual things happening to the aforementioned operators.
Sorry but I didn't remember exactly what was my problem with memory macros. My memory manager doesn't override any new/delete itself either, it has a simple void* alloc(size_t) and a void dealloc(void*) function. The problems appeared at places where I tried to use it.

My problem is related to my string implementation where I need to use placement new to write things down properly. Basically I allocate a single refcounted memory buffer with both data and string character array in a single allocation, and use placement new on this buffer. The memory macros made this impossible.

Also, to simplify usage of the optimized memory manager I would make a simple base class called FastAllocated which overriden new/delete operators, and subclass from it where needed. Using factories one would need more code and/or macros to have similar functionality.

Factories (recyclers) use a different approach than mine. I don't like to use them for recycling because objects of the same size can't usually share the same recycler instance, and it's more problematic to use them for a lot of classes. There are places where their use them for extra performance, like for Billboards in Ogre.
rincewind
Kobold
Posts: 33
Joined: Tue Jan 04, 2005 5:57 pm

Post by rincewind »

Hi,

just stumbled across the same problem with boost::serialization. Did anybody ever find a solutions (short of disabling the memory manager).?
I've been clutterning my code already with "OgreNoMemoryMacros.h" but it doesn't really help.

I really like to have the memory manager to detect leaks.

Rincewind
User avatar
BenO
Goblin
Posts: 241
Joined: Mon Apr 18, 2005 5:03 pm

Post by BenO »

bump :o

i have the same pb . anyone got a solution ?
Benjamin RIGAUD
Software Engineer
User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
x 179
Contact:

Post by jacmoe »

BenO wrote:i have the same pb . anyone got a solution ?
A problem with boost::serialization ?
Will try and get it to work. I am quite interested in using that library myself. :wink:
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
User avatar
BenO
Goblin
Posts: 241
Joined: Mon Apr 18, 2005 5:03 pm

Post by BenO »

yeah :) boost::serialization.

i disabled the ogre memory manager : and it works fine.

But i would be happy to debug my application with serialization
and leak detection :o


Ps: if u plan on using it, i suggest u download the CVS version :
The last release is quite old.
Benjamin RIGAUD
Software Engineer
User avatar
:wumpus:
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3067
Joined: Tue Feb 10, 2004 12:53 pm
Location: The Netherlands
x 1

Post by :wumpus: »

I generally disable the Ogre memory manager as well nowadays, it interferes with almost every other library (Sh, Boost, ...) due to the wacky free/alloc macro definitions.
I'm sure there is a better way to do memory debugging than having to compile everything with some macros.
User avatar
BenO
Goblin
Posts: 241
Joined: Mon Apr 18, 2005 5:03 pm

Post by BenO »

good point :o
Benjamin RIGAUD
Software Engineer
User avatar
Game_Ender
Ogre Magi
Posts: 1269
Joined: Wed May 25, 2005 2:31 am
Location: Rockville, MD, USA

Post by Game_Ender »

:wumpus: wrote: I'm sure there is a better way to do memory debugging than having to compile everything with some macros.
It seems like I have been pimping this program a lot lately, but if you develope on linux you could just run your program through valgrind. It has does has done a very good job of catching memory leaks for me in the past. It catches things like using uninitialized values and other little memory errors that cause undefined behavior. The only problem is it might make Ogre so slow that you wouldn't be able to test.
getli
Gnoblar
Posts: 19
Joined: Sun Oct 03, 2004 2:59 pm

Post by getli »

Bump. Ran into the same problem with boost serialization

Code: Select all

#include "OgreNoMemoryMacros.h"
.. boost
#include "OgreMemoryMacros.h" 
doesn't work. Including before the Ogre includes doesn't either. Has anyone solved this? (Disabling the memory manager works, but is not desirable in debug builds for me).
User avatar
xavier
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 9481
Joined: Fri Feb 18, 2005 2:03 am
Location: Dublin, CA, US
x 22

Post by xavier »

Never did. I ended up "rolling-our-own" serialization solution for the classes in question.
getli
Gnoblar
Posts: 19
Joined: Sun Oct 03, 2004 2:59 pm

Post by getli »

That's a shame. Boost serialisation is easy to use and well engineered. I don't want to reinvent the wheel, but also don't want to loose the leak checking. Could anyone of the Ogre team look into that? Surely there must be a way to make the memory manager more compatible with other libraries. One radical way I've used in the past would be replacing new by something like OGRE_NEW, so only allocations done with OGRE_NEW/OGRE_MALLOC would be tracked. This way you can be sure that external libraries work fine. They don't get tracked but IMO that's a small tradeoff. I know this change would be a big one, but might be the cleanest solution to solve this problem (I am sure boost serialisation is not the only problem case as overloading/defining global new and delete is normally not encouraged).
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Post by Praetor »

Yeah, I'm planning on using serialization as well, but I haven't yet run into problems. To be fair the serialization happens at a level above Ogre (in a wrapping engine) so it could help. I'll probably grab the source release of Ogre soon and compile it without the memory manager on to avoid any problems. I'll be using an external app (like valgrind) to catch memory leaks anyway.

I don't think there's any way around it without a lot of work. You'd have to override the new/delete operators for EVERY class in Ogre instead of once for the global ones to make the memory manager Ogre-specific, and even then it might interfere with things.
getli
Gnoblar
Posts: 19
Joined: Sun Oct 03, 2004 2:59 pm

Post by getli »

I wasn't talking about overloading the member new/delete operators, but more about doing smth like this:

Code: Select all

//------------------------------------------------------------------------------
void* operator new(size_t size, const char* file, unsigned int line)
//------------------------------------------------------------------------------
{
    return _malloc_dbg(size, _NORMAL_BLOCK, file, line);
}

//------------------------------------------------------------------------------
void operator delete(void* pointer, const char*, unsigned int)
//------------------------------------------------------------------------------
{
    _free_dbg(pointer, _NORMAL_BLOCK);
}

#ifndef NDEBUG
#   define OGRE_NEW new(__FILE__, __LINE__)
#   define OGRE_DELETE delete
#   define OGRE_DELETE_A delete[]
#else // !NDEBUG
#   define OGRE_NEW new
#   define OGRE_DELETE delete
#   define OGRE_DELETE_A delete[]
#endif // !NDEBUG
Then you'd simply do a replace in files with new -> OGRE_NEW, delete -> OGRE_DELETE. As a client you'd use new for untracked and OGRE_NEW for tracked memory. Nice and clean and does not interfere with external libraries at all. If you want to track STL or smth similar you simply provide custom allocators that use OGRE_NEW instead of new
OvermindDL1
Gnome
Posts: 333
Joined: Sun Sep 25, 2005 7:55 pm

Post by OvermindDL1 »

I've been having a lot of issues (and I mean a metric ton) with OGRE's memory manager and other libraries recently. I've just started seperating out my engine code from graphic code in full, fixes that right up, lot more coding though.
grizzley90
Gnome
Posts: 318
Joined: Wed Jan 18, 2006 7:22 pm
Location: Montreal, Canada

Post by grizzley90 »

Has this problem ever been solved? Or is the best way to simply just remove the ogre memory manager? :?
Post Reply