background loading problems with Zip Archive

A place for users of OGRE to discuss ideas and experiences of utilitising OGRE in their games / demos / applications.
Post Reply
User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany

background loading problems with Zip Archive

Post by tdev » Sun May 02, 2010 9:27 am

so, i am currently trying to add background loading to our project (http://rigsofrods.com) and i ran into some problems.

first, the facts, like our ogre configuration:

Code: Select all

/* #undef OGRE_STATIC_LIB */
#define OGRE_CONFIG_LITTLE_ENDIAN
/* #undef OGRE_CONFIG_BIG_ENDIAN */
#define OGRE_DOUBLE_PRECISION 0
#define OGRE_MEMORY_ALLOCATOR 4
#define OGRE_CONTAINERS_USE_CUSTOM_MEMORY_ALLOCATOR 1
#define OGRE_STRING_USE_CUSTOM_MEMORY_ALLOCATOR 0
#define OGRE_MEMORY_TRACKER_DEBUG_MODE 0
#define OGRE_MEMORY_TRACKER_RELEASE_MODE 0
#define OGRE_THREAD_SUPPORT 2
#define OGRE_THREAD_PROVIDER 1
#define OGRE_NO_FREEIMAGE 0
#define OGRE_NO_DDS_CODEC 0
#define OGRE_NO_PVRTC_CODEC 1
#define OGRE_NO_ZIP_ARCHIVE 0
#define OGRE_NO_VIEWPORT_ORIENTATIONMODE 1
#define OGRE_USE_NEW_COMPILERS 1
#define OGRE_USE_BOOST 1
testing under windows 7 with VS2008 and Ogre 1.7.0.

so our small background loading test class:

Code: Select all

/*
This source file is part of Rigs of Rods
Copyright 2005,2006,2007,2008,2009 Pierre-Michel Ricordel
Copyright 2007,2008,2009 Thomas Fischer

For more information, see http://www.rigsofrods.com/

Rigs of Rods is free software: you can redistribute it and/or modify
it under the terms of the GNU General Public License version 3, as
published by the Free Software Foundation.

Rigs of Rods is distributed in the hope that it will be useful,
but WITHOUT ANY WARRANTY; without even the implied warranty of
MERCHANTABILITY or FITNESS FOR A PARTICULAR PURPOSE.  See the
GNU General Public License for more details.

You should have received a copy of the GNU General Public License
along with Rigs of Rods.  If not, see <http://www.gnu.org/licenses/>.
*/

// created by Thomas Fischer, 1st of May 2010

#ifndef MESHOBJECT_H__
#define MESHOBJECT_H__

#include "Ogre.h"
#include "rormemory.h"

class MeshObject : public Ogre::ResourceBackgroundQueue::Listener, public Ogre::Resource::Listener
{
public:
	MeshObject(Ogre::SceneManager *smgr, Ogre::String meshName, Ogre::String entityName, Ogre::SceneNode *sceneNode=0, bool backgroundLoading=true) : \
		smgr(smgr), entityName(entityName), meshName(meshName), sceneNode(sceneNode), ent(0), backgroundLoading(backgroundLoading), loaded(false)
	{
		// create a new sceneNode if not existing
		if(!sceneNode)
			sceneNode = smgr->getRootSceneNode()->createChildSceneNode();

		loadMesh();
	}

	~MeshObject()
	{
       if(backgroundLoading && !mesh.isNull())
           mesh->unload();
	}

protected:
	Ogre::SceneManager *smgr;
	Ogre::String meshName, entityName;
	Ogre::SceneNode *sceneNode;
	Ogre::Entity *ent;
	Ogre::MeshPtr mesh;
	Ogre::BackgroundProcessTicket ticket;
	bool backgroundLoading;
	bool loaded;
	
	void postProcess()
	{
		loaded=true;
		sceneNode->setVisible(true);
	}

	void loadMesh()
	{
		Ogre::String resourceGroup = Ogre::ResourceGroupManager::DEFAULT_RESOURCE_GROUP_NAME;
		Ogre::MeshPtr mesh;
		mesh = static_cast<Ogre::MeshPtr>(Ogre::MeshManager::getSingletonPtr()->create(meshName, resourceGroup));
		if(backgroundLoading)
		{
			mesh->setBackgroundLoaded(true);
			mesh->addListener(this);

			ticket = Ogre::ResourceBackgroundQueue::getSingleton().load(
				Ogre::MeshManager::getSingletonPtr()->getResourceType(), 
				mesh->getName(), 
				resourceGroup,
				false,
				0,
				0,
				0);

			//Ogre::Root::getSingleton().addFrameListener(this);
			// now create an entity around the mesh and attach it to the scene graph
			try
			{
				if(entityName.empty())
					ent = smgr->createEntity(meshName);
				else
					ent = smgr->createEntity(entityName, meshName);
				if(ent)
					sceneNode->attachObject(ent);
			} catch(Ogre::Exception& e)
			{
				LogManager::getSingleton().logMessage("error loading mesh: " + meshName + ": " + e.getFullDescription());
				return;
			}

		} else
		{
			// now create an entity around the mesh and attach it to the scene graph
			try
			{
				if(entityName.empty())
					ent = smgr->createEntity(meshName);
				else
					ent = smgr->createEntity(entityName, meshName);
				if(ent)
					sceneNode->attachObject(ent);
			} catch(Ogre::Exception& e)
			{
				LogManager::getSingleton().logMessage("error loading mesh: " + meshName + ": " + e.getFullDescription());
				return;
			}
			postProcess();
		}
	}

	void operationCompleted(BackgroundProcessTicket ticket, const BackgroundProcessResult& result)
	{
		LogManager::getSingleton().logMessage("operationCompleted: " + meshName);
		if(ticket == this->ticket)
			postProcess();
	}

	void backgroundLoadingComplete(Resource *r)
	{
		// deprecated, use loadingComplete instead
	}

	void backgroundPreparingComplete(Resource *r)
	{
		// deprecated, use preparingComplete instead
	}

	void loadingComplete(Resource *r)
	{
		LogManager::getSingleton().logMessage("loadingComplete: " + meshName);
		postProcess();
	}

	void preparingComplete(Resource *r)
	{
		LogManager::getSingleton().logMessage("preparingComplete: " + meshName);
	}

	void unloadingComplete(Resource *r)
	{
		LogManager::getSingleton().logMessage("unloadingComplete: " + meshName);
	}
};

#endif //MESHOBJECT_H__
and its usage:

Code: Select all

			props[free_prop].snode = manager->getRootSceneNode()->createChildSceneNode();
			MeshObject *mo = new MeshObject(manager, meshname, "", props[free_prop].snode, enable_background_loading);

so far so good, works pretty well if you load zips from the "FileSystem". But once its residing inside a zip i get only half of the meshes displayed:

Code: Select all

10:41:59: Mesh: Loading ad28-1.mesh.
10:41:59: OGRE EXCEPTION(7:InternalErrorException): ad28-1.mesh - error from zziplib: data error in ZipDataStream::read at ..\..\OgreMain\src\OgreZip.cpp (line 315)
10:41:59: Mesh: Loading vezahydro.mesh.
10:41:59: Mesh: Loading veza.mesh.
10:41:59: D3D9 : Loading 2D Texture, image name : 'beaconflare.dds' with 5 mip map levels
10:41:59: OGRE EXCEPTION(7:InternalErrorException): veza.mesh - error from zziplib: buffer error in ZipDataStream::read at ..\..\OgreMain\src\OgreZip.cpp (line 315)
10:41:59: Mesh: Loading vezahydro2.mesh.
10:41:59: Mesh: Loading rameno1.mesh.
10:41:59: Mesh: Loading rameno2.mesh.
10:41:59: Mesh: Loading rameno3.mesh.
i tested that the archive is not corrupt and it loads very fine in non-background loading mode.

so to my question:
a) why is zziplib not working with threading? isnt it supposed to be thread safe?
b) how to load the meshes textures in background as well?

thanks in advance for helping!
0 x

User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany

Re: background loading problems with Zip Archive

Post by tdev » Sun May 02, 2010 1:26 pm

0 x

psquare
Hobgoblin
Posts: 554
Joined: Tue Nov 14, 2006 3:26 pm

Re: background loading problems with Zip Archive

Post by psquare » Mon Feb 21, 2011 1:34 pm

Sorry for resurrecting this old thread, but I am having the exact same problem:

In threaded mode
-Loading resources when they are not in an archive works great
-Loading resources when they are in an archive results in the above error.

I could not find any other information on this.

@tdev: If you worked around this somehow, any pointers in that direction would really help.
0 x

psquare
Hobgoblin
Posts: 554
Joined: Tue Nov 14, 2006 3:26 pm

Re: background loading problems with Zip Archive

Post by psquare » Tue Feb 22, 2011 6:26 am

I don't think the Zip file operations are thread safe. Upon debugging it seems to me that concurrent reads are happening on the stream via multiple threads. See images/call stacks of the 2 different threads.

Image Image
0 x

User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany

Re: background loading problems with Zip Archive

Post by tdev » Fri Feb 25, 2011 10:44 pm

psquare wrote:Sorry for resurrecting this old thread, but I am having the exact same problem:

In threaded mode
-Loading resources when they are not in an archive works great
-Loading resources when they are in an archive results in the above error.

I could not find any other information on this.

@tdev: If you worked around this somehow, any pointers in that direction would really help.
haven't worked around this.

only solution that i saw after investigating this is to implement your own ZipArchive which is then threadsafe. But then you will need to find thread-safe 3rd party zip libraries again ... :-\
0 x

psquare
Hobgoblin
Posts: 554
Joined: Tue Nov 14, 2006 3:26 pm

Re: background loading problems with Zip Archive

Post by psquare » Sat Feb 26, 2011 2:52 am

only solution that i saw after investigating this is to implement your own ZipArchive which is then threadsafe.
Yes. That's exactly what I ended up doing.

I needed the ability to update the zip file as well (overwrite or add new files), so I needed this anyway. I used zlib, zziplib and minizip.
0 x

User avatar
tdev
Silver Sponsor
Silver Sponsor
Posts: 244
Joined: Thu Apr 12, 2007 9:21 pm
Location: Germany

Re: background loading problems with Zip Archive

Post by tdev » Wed Mar 23, 2011 4:12 pm

psquare wrote:
only solution that i saw after investigating this is to implement your own ZipArchive which is then threadsafe.
Yes. That's exactly what I ended up doing.

I needed the ability to update the zip file as well (overwrite or add new files), so I needed this anyway. I used zlib, zziplib and minizip.
thanks for the feedback :)
0 x

SufferinPup
Greenskin
Posts: 119
Joined: Tue Mar 02, 2010 11:09 pm
x 2

Re: background loading problems with Zip Archive

Post by SufferinPup » Mon Aug 25, 2014 8:07 pm

Sorry to dig up an old thread but I'm wondering if this is still a problem in Ogre 1.9? We do a lot of mid-game background loading, which works fine using FileSystem but if I put stuff in zip files we get a lot of crashing. I find it odd that Ogre supports background asset loading, but the only asset archive it supports is not thread safe. Is the only solution still to create your own archive using a different zip library? I'm hoping I just missed a compile option. :)

Thanks for any help!
0 x

Post Reply