Compiling on g++ 4.4/4.5

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
Post Reply
User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Compiling on g++ 4.4/4.5

Post by sparkprime » Sat Nov 26, 2011 7:22 pm

I noticed a while ago that a g++ caused buggy codegen when building ogre with g++ 4.4 and 4.5.

Now, the latest version of ubuntu no-longer ships with older g++ compilers so I've been forced to actually face this problem.

I've found that adding at std::cout << "" in the middle of Skeleton::readBoneParent(), just before the getBone() calls, avoids the problem, presumably because it adds enough stuff to the code that whatever assumptions the compiler is using to trigger an 'optimisation' no longer hold, because it doesn't understand the extra code added, and that is causing e.g. variables to be considered live instead of dead, or whatever. It's a fragile solution but I can't think of anything better at this point.

Curiously, printf("") doesn't work because it simply gets inlined, unrolled, and eliminated as dead code. free(NULL) didn't do any good either, even though it is present in the generated code. printf("\n") works, but that actually prints something which is annoying.

The actual symptom is spurious load failures of skeletons. It is caused by an exception thrown inside Node::addChild(), due presumably to bogus parameters, which is then caught in Mesh::setSkeletonName and unhelpfully replaced with a generic error message.

Code: Select all

            OGRE_EXCEPT(Exception::ERR_INVALIDPARAMS,
                "Node '" + child->getName() + "' already was a child of '" +
                child->mParent->getName() + "'.",
                "Node::addChild");

Code: Select all

"Unable to load skeleton vehicles/Evo/Body.skeleton for Mesh vehicles/Evo/Body.mesh. This Mesh will not be animated. You can ignore this message if you are using an offline tool."
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sat Nov 26, 2011 7:28 pm

It could also be some irritating timing-sensitive concurrency bug, and adding this bit of overhead 'fixes' it. However I'm not doing anything in a background thread at this point during execution.
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sat Nov 26, 2011 7:33 pm

The generated ASM is identical except for the insertion of the indented instructions below, which is totally inconsistent with my explanation of it being an optimisation bug (although it only occurs with high -O levels). So I dunno what's going on.

Code: Select all

 85437da:   55                      push   %ebp
 85437db:   89 e5                   mov    %esp,%ebp
 85437dd:   57                      push   %edi
 85437de:   56                      push   %esi
 85437df:   53                      push   %ebx
 85437e0:   83 ec 2c                sub    $0x2c,%esp
 85437e3:   8b 75 08                mov    0x8(%ebp),%esi
 85437e6:   8b 7d 0c                mov    0xc(%ebp),%edi
 85437e9:   8b 5d 10                mov    0x10(%ebp),%ebx
 85437ec:   c7 44 24 0c 01 00 00    movl   $0x1,0xc(%esp)
 85437f3:   00 
 85437f4:   8d 4d e6                lea    -0x1a(%ebp),%ecx
 85437f7:   89 4c 24 08             mov    %ecx,0x8(%esp)
 85437fb:   89 7c 24 04             mov    %edi,0x4(%esp)
 85437ff:   89 34 24                mov    %esi,(%esp)
 8543802:   e8 55 65 e6 ff          call   83a9d5c <_ZN4Ogre10Serializer10readShortsERNS_9SharedPtrINS_10DataStreamEEEPtj>
 8543807:   c7 44 24 0c 01 00 00    movl   $0x1,0xc(%esp)
 854380e:   00 
 854380f:   8d 45 e4                lea    -0x1c(%ebp),%eax
 8543812:   89 44 24 08             mov    %eax,0x8(%esp)
 8543816:   89 7c 24 04             mov    %edi,0x4(%esp)
 854381a:   89 34 24                mov    %esi,(%esp)
 854381d:   e8 3a 65 e6 ff          call   83a9d5c <_ZN4Ogre10Serializer10readShortsERNS_9SharedPtrINS_10DataStreamEEEPtj>
                                 8543822:   c7 44 24 08 00 00 00    movl   $0x0,0x8(%esp)
                                 8543829:   00
                                 854382a:   c7 44 24 04 f7 fb 99    movl   $0x899fbf7,0x4(%esp)
                                 8543831:   08
                                 8543832:   c7 04 24 40 7c ba 09    movl   $0x9ba7c40,(%esp)
                                 8543839:   e8 46 e7 b0 ff          call   8051f84 <_ZSt16__ostream_insertIcSt11char_traitsIcEERSt13basic_ostreamIT_T0_ES6_PKS3_i@plt>
 854383e:   8b 3b                   mov    (%ebx),%edi
 8543840:   0f b7 55 e4             movzwl -0x1c(%ebp),%edx
 8543844:   89 54 24 04             mov    %edx,0x4(%esp)
 8543848:   89 1c 24                mov    %ebx,(%esp)
 854384b:   ff 97 d0 00 00 00       call   *0xd0(%edi)
 8543851:   89 c6                   mov    %eax,%esi
 8543853:   8b 03                   mov    (%ebx),%eax
 8543855:   0f b7 4d e6             movzwl -0x1a(%ebp),%ecx
 8543859:   89 4c 24 04             mov    %ecx,0x4(%esp)
 854385d:   89 1c 24                mov    %ebx,(%esp)
 8543860:   ff 90 d0 00 00 00       call   *0xd0(%eax)
 8543866:   8b 16                   mov    (%esi),%edx
 8543868:   89 44 24 04             mov    %eax,0x4(%esp)
 854386c:   89 34 24                mov    %esi,(%esp)
 854386f:   ff 92 90 00 00 00       call   *0x90(%edx)
 8543875:   83 c4 2c                add    $0x2c,%esp
 8543878:   5b                      pop    %ebx
 8543879:   5e                      pop    %esi
 854387a:   5f                      pop    %edi
 854387b:   5d                      pop    %ebp
 854387c:   c3                      ret    
 854387d:   90                      nop    
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Compiling on g++ 4.4/4.5

Post by CABAListic » Sat Nov 26, 2011 9:12 pm

I haven't seen this error so far. Is there any way to reproduce it with the SampleBrowser?
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sun Nov 27, 2011 1:57 pm

The skeletal animation demo works, so no, I don't think so. However I don't use the standard build system when I build my app, and I have no idea what optimisation flags or gcc version are used when building the sample browser.

In my build system I use -DNDEBUG -O2 -finline-functions -funroll-loops

When building with cmake i just run 'make'
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sun Nov 27, 2011 2:24 pm

Could also be that my particular skeleton file is tickling the code in a particular way that the blue ninja girl skeleton doesn't
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sun Nov 27, 2011 3:57 pm

OK I've found out why adding the cout fixes it -- it causes the function to not be inlined within SkeletonSerializer::importSkeleton

so my examination of the generated ASM for this function in the two cases was pontless, since in the original case the ASM i was reading was not actually being called...

quite why inlining it causes a problem I have not determined but presumably it enables some other optimisation within importSkeleton which is the cause
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Compiling on g++ 4.4/4.5

Post by CABAListic » Sun Nov 27, 2011 4:19 pm

Cmake (unless explicitly told differently) will use the standard compiler on your system, i.e. whichever version responds to calling g++. On my Ubuntu systems this has been >= 4.4 for a while, and I haven't seen any problems so far. That's why a simple example of the problem would be helpful.

I assume there might be a non-standard keyword to tell GCC not to inline a certain function, which would definitely be a better 'fix' than the cout hack. In all honesty, though, I'd rather we get to the actual root of the problem.
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sun Nov 27, 2011 6:30 pm

Is ogre usually compiled with -finline-functions ?

If not, this bug would not arise.
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Compiling on g++ 4.4/4.5

Post by CABAListic » Sun Nov 27, 2011 7:23 pm

I'm not entirely sure. I'm quite certain we don't add that flag, but I don't know if it's included in CMake's default release flags.
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sun Nov 27, 2011 7:34 pm

-finline-functions will cause functions to be inlined when they don't have an explicit inline keyword (and i think definitions in class bodies are equivalent to having an inline keyword). It only inlines things that it thinks are a good idea based on its (imperfect) heuristics, hence it is not on by default at -O2.

I usually turn it on because otherwise you end up putting 'inline' everywhere in order to get performance.

How do I modify cmake to add this flag?
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Compiling on g++ 4.4/4.5

Post by CABAListic » Sun Nov 27, 2011 7:47 pm

There's a CMake variable for that, called CMAKE_CXX_FLAGS. I think it might also pick up the standard environment variable CXXFLAGS.
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sun Nov 27, 2011 8:35 pm

Ok I tried doing this but couldn't reproduce the problem with the sample browser

make "CMAKE_CXX_FLAGS=-O2 -finline-functions"
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Sun Nov 27, 2011 8:36 pm

Is there some way I can get it to actually show me the g++ commandlines?
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Compiling on g++ 4.4/4.5

Post by CABAListic » Sun Nov 27, 2011 9:26 pm

Eh, no, it's a CMake variable; you need to modify it within CMake, before running. Best install and use cmake-gui to have easy access to the variables.
There's also one to show the full command lines, something like CMAKE_VERBOSE_MAKEFILES, I think.
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Mon Nov 28, 2011 12:10 pm

O.K. I can load jaiqua.mesh and jaiqua.skeleton fine in all cases, but my own skeletons only work with inlining disabled. These skeletons were produced from the blender export tool and OgreXMLConverter.exe. They're not even particularly complex skeletons -- it's just a car where the body is the root bone, and the suspension linkage + wheels are bones as well. I suspect it's something asinine like the bones being listed in a different order in the binary file, or something like that, that triggers this.
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Mon Nov 28, 2011 1:21 pm

I also can't get it to show up with the default ogre build at all -- this could be anything, e.g. the fact it's a shared library with -fPIC, to well... anything

However I do get a valgrind error very close to the offending code in both cases. They vary a bit, but what seems to be happening is that there is *always* UB going on, but sometimes we are lucky and it doesn't result in a visible bug.

We are lucky if any of the following are true

* gcc < 4.4
* skeleton file on disk has certain properties
* certain build options are used (or __attribute__((noinline)))

There is always an "Conditional jump or move depends on uninitialised value(s)" error but it appears in different places depending on the input file. For my skeleton it shows up when doing readChunk in importSkeleton. However for the jaiqua it appears in readAnimationTrack. Perhaps there is a buffer overrun on the input buffer used to spool data from disk, or something like that? I.e. more bytes have been read from the buffer by the skeleton reading code, than were written into it from the file? That would cause different behaviour depending on the size of the file.

Can you see any errors from valgrind when loading a skeleton?
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Mon Nov 28, 2011 1:32 pm

The valgrind errors even appear with no optimisation, so I suspect this is not a gcc bug after all.
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Compiling on g++ 4.4/4.5

Post by CABAListic » Mon Nov 28, 2011 3:00 pm

Gotta hate these kinds of bugs. I'll try to find some time to take a look at it (before the final release, anyway), although I don't think I can do any better than what you already tried.

In any case, a few questions:

1. I figure from your comments that you are probably building statically? Could you tell us more about your actual setup, i.e. your dev system (32bit/64bit etc.) and the way you build Ogre?
2. Since you mentioned concurrency, have you tried to disable the Ogre threading support?
3. Are you using some kind of Unity build technique (i.e. collecting all Ogre files into a single or very few compile units to speed up compile times)? Our CMake system supports this now, but I faced a few obscure bugs in the beginning thanks to that technique. Just want to make sure it's not related to something like that.
4. Have you perchance tested on Windows, ideally even with a recent MinGW? Any problems there?
0 x

User avatar
sparkprime
Ogre Magi
Posts: 1137
Joined: Mon May 07, 2007 3:43 am
Location: Ossining, New York
Contact:

Re: Compiling on g++ 4.4/4.5

Post by sparkprime » Mon Nov 28, 2011 4:55 pm

CABAListic wrote: 1. I figure from your comments that you are probably building statically? Could you tell us more about your actual setup, i.e. your dev system (32bit/64bit etc.) and the way you build Ogre?
Yes, I build everything (including relevant RenderSystems and plugins) into a single .a file that I add in with other .a files and my own .o files to produce my final executable.

The makefile is here http://gritengine.svn.sourceforge.net/v ... iew=markup
2. Since you mentioned concurrency, have you tried to disable the Ogre threading support?
No, in fact I have additional threads in my code doing background streaming in a more sophisticated way than Ogre does -- I am not using Ogre::BackgroundResourceLoader or whatever it's called.

However at the time of doing the skeleton loading, my background thread is idle because I'm using a 'map' that is a simple test and is fully loaded, not streamed.

Additionally, I have done tests with code that looks like this, and there shouldn't be any race condition there

Code: Select all

#include <cstdlib>

#include <Ogre.h>
#include <OgreFontManager.h>
#include <OgreMeshManager.h>
#include <OgreOverlayElementFactory.h>
#include <OgreVector3.h>
#include <OgreQuaternion.h>
#include <OgreCustomCompositionPass.h>
#include <OgreCompositor.h>
#ifdef NO_PLUGINS
#  include <OgreOctreeSceneManager.h>
#  include "OgreOctreePlugin.h"
#  include "OgreGLPlugin.h"
#  include "OgreCgPlugin.h"
#  ifdef WIN32
#    include "OgreD3D9Plugin.h"
#  endif
#endif

bool use_hwgamma = false; //getenv("GRIT_NOHWGAMMA")==NULL;

Ogre::Root *ogre_root;
Ogre::SceneManager *ogre_sm;
Ogre::SceneNode *ogre_root_node;
Ogre::Camera *left_eye;
Ogre::Camera *right_eye;
Ogre::Light *ogre_sun;
Ogre::SceneNode *ogre_celestial;
Ogre::SceneNode *ogre_sky_node;
Ogre::Entity *ogre_sky_ent;
Ogre::Viewport *overlay_vp;
Ogre::Viewport *left_vp;
Ogre::Viewport *right_vp;
Ogre::TexturePtr anaglyph_fb;
Ogre::RenderWindow *ogre_win;
#ifdef NO_PLUGINS
    Ogre::GLPlugin *gl;
    Ogre::OctreePlugin *octree;
    //Ogre::ParticleFXPlugin *pfx;
    Ogre::CgPlugin *cg;
    #ifdef WIN32
        Ogre::D3D9Plugin *d3d9;
    #endif
#endif

int main(int argc, char **argv)
{
    if (argc!=2) {
        fprintf(stderr, "Usage: %s blah.mesh\n", argv[0]);
        exit(EXIT_FAILURE);
    }
    std::string mesh_name = argv[1];
    try {

        #ifdef WIN32
        bool use_d3d9 = getenv("GRIT_GL")==NULL;
        #else
        bool use_d3d9 = false;
        #endif

        #ifdef NO_PLUGINS
            ogre_root = OGRE_NEW Ogre::Root("","","");

            gl = OGRE_NEW Ogre::GLPlugin();
            ogre_root->installPlugin(gl);

            octree = OGRE_NEW Ogre::OctreePlugin();
            ogre_root->installPlugin(octree);

            cg = OGRE_NEW Ogre::CgPlugin();
            ogre_root->installPlugin(cg);

            #ifdef WIN32
            if (use_d3d9) {
                Ogre::D3D9Plugin *d3d9 = OGRE_NEW Ogre::D3D9Plugin();
                ogre_root->installPlugin(d3d9);
            }
            #endif
        #else
            ogre_root = OGRE_NEW Ogre::Root("plugins.cfg","","");
        #endif


        Ogre::RenderSystem *rs;
        if (use_d3d9) {
            rs = ogre_root->getRenderSystemByName("Direct3D9 Rendering Subsystem");
            rs->setConfigOption("Allow NVPerfHUD","Yes");
            rs->setConfigOption("Floating-point mode","Consistent");
            rs->setConfigOption("Video Mode","800 x 600 @ 32-bit colour");
        } else {
            rs = ogre_root->getRenderSystemByName("OpenGL Rendering Subsystem");
            rs->setConfigOption("RTT Preferred Mode","FBO");
            rs->setConfigOption("Video Mode","800 x 600");
        }
        rs->setConfigOption("sRGB Gamma Conversion",use_hwgamma?"Yes":"No");
        rs->setConfigOption("Full Screen","No");
        rs->setConfigOption("VSync","Yes");
        ogre_root->setRenderSystem(rs);

        ogre_root->initialise(true,"Grit Game Window");

        ogre_win = ogre_root->getAutoCreatedWindow();



        Ogre::ResourceGroupManager::getSingleton().addResourceLocation(".","FileSystem","General",false);
        Ogre::ResourceGroupManager::getSingleton().initialiseAllResourceGroups();
        Ogre::MeshManager::getSingleton().load(mesh_name,"General");



        if (ogre_sm && ogre_root) ogre_root->destroySceneManager(ogre_sm);
        if (ogre_root) OGRE_DELETE ogre_root;
        #ifdef NO_PLUGINS
            OGRE_DELETE gl;
            OGRE_DELETE octree;
            OGRE_DELETE cg;
            #ifdef WIN32
                OGRE_DELETE d3d9;
            #endif
        #endif


        return EXIT_SUCCESS;

    } catch( Ogre::Exception& e ) {
            std::cerr << "An exception has occured: " << e.getFullDescription().c_str() << std::endl;
    }

}
3. Are you using some kind of Unity build technique (i.e. collecting all Ogre files into a single or very few compile units to speed up compile times)? Our CMake system supports this now, but I faced a few obscure bugs in the beginning thanks to that technique. Just want to make sure it's not related to something like that.
No haven't tried doing that with Ogre yet.
4. Have you perchance tested on Windows, ideally even with a recent MinGW? Any problems there?
This bug has existed for a long time, at least a year I think, probably longer, and I have avoided it by using an older gcc until now.

I have tested on windows during this timeframe (although not in the last few months) as this is the primary platform, and this bug does not occur with Visual Studio 2008 Express, but I haven't tried MingW for many years because (pre 4.0) its performance was inferior to MSVC so I gave up on it.

I think Linux is the place to debug this because you have valgrind there, and that seems like the most reliable way to demonstrate that the bug exists with a given set of code.
0 x

visor24
Gnoblar
Posts: 16
Joined: Mon Mar 12, 2012 2:21 pm

Re: Compiling on g++ 4.4/4.5

Post by visor24 » Fri Mar 01, 2013 7:45 am

Same problem here.
We are trying to port our game into Android platform and suffer exactly same problem here.
We worked around this with same 'cout hack' above but it would nice if there are any other elegant solutions.
We are using ndk r8d and clang 3.1, by the way.

Thanks.
0 x

visor24
Gnoblar
Posts: 16
Joined: Mon Mar 12, 2012 2:21 pm

Re: Compiling on g++ 4.4/4.5

Post by visor24 » Fri Mar 01, 2013 9:01 am

We are using Ogre 1.8, by the way.
0 x

Post Reply