Race condition in ZipDataStream::read

[Hilarius86]

When I try to load two meshes using the same texture, the ZipDataStream::read fails with "data error". If I load each file individually or with sufficent time in between, both models load just fine. The error points towards a race condition. I work with 2.2.7 (Windows10 / Dx11) and already backported one fix (84c80a5). Sadly, the error persists.

My first guess was the missing locks in the ZipDataStream class and its cache usage, but after adding locks, I still get the exception and realize that I seem to have two distinct ZipArchive objects / ZipDataStreams managing the same zziplib handle.

Any advise on how to fix the problem?

I can provide the offending models, but a two random low poly, big texture files should suffice.
The following code can be used to provoke the exceptions:

Code: Select all

Ogre::SceneManager *sceneManager = mGraphicsSystem->getSceneManager();

std::vector<Ogre::String> MeshNames = { "VZ244.3_G2.mesh", "VZ244.3_G3.mesh" };
Ogre::String GroupName = "Popular";
for (int i = 0; i < MeshNames.size(); ++i)
{
    Ogre::MeshPtr Mesh = Ogre::MeshManager::getSingleton().load( MeshNames[i], GroupName );

Ogre::Item *item = sceneManager->createItem(Mesh, Ogre::SCENE_DYNAMIC );
Ogre::SceneNode *sceneNode = sceneManager->getRootSceneNode( Ogre::SCENE_DYNAMIC )
                                 ->createChildSceneNode( Ogre::SCENE_DYNAMIC );
sceneNode->attachObject( item );
sceneNode->setPosition( i, 0.f, 0.f );
sceneNode->scale( 1.f, 1.f, 1.f );
}

Code: Select all

16:07:52: Mesh: Loading VZ244.3_G2.mesh.
16:07:52: Corrupted chunk detected! Stream name: 'VZ244.3_G2.mesh' Chunk id: 12800
16:07:52: Corrupted chunk detected! Stream name: 'VZ244.3_G2.mesh' Chunk id: 3947
16:07:52: Corrupted chunk detected! Stream name: 'VZ244.3_G2.mesh' Chunk id: 26746
16:07:52: Mesh: Loading VZ244.3_G3.mesh.
16:07:55: OGRE EXCEPTION(7:InternalErrorException): Col.jpg - error from zziplib: data error in ZipDataStream::read at C:\pathX\ogre-next\OgreMain\src\OgreZip.cpp (line 377)
16:07:56: OGRE EXCEPTION(7:InternalErrorException): VZ244.3_G3.mesh - error from zziplib: data error in ZipDataStream::read at C:\pathX\ogre-next\OgreMain\src\OgreZip.cpp (line 377)
16:07:56: OGRE EXCEPTION(7:InternalErrorException): Col.jpg - error from zziplib: data error in ZipDataStream::read at C:\pathX\ogre-next\OgreMain\src\OgreZip.cpp (line 377)

[dark_sylinc]

Hi.

It sounds like you're creating Items from multiple threads, which is not supported.

Cheers
Matias

[Hilarius86]

The code example, though not explicitly stated, was run from the createScene01 method of a sample (just reused V2MeshGameState). So all the invocations of createItem are from the main thread I believe.

The multithreading comes into place as the texture is loaded in the background.
The crash happens sometimes in the main thread when trying to load the second mesh. Some times also in the worker thread streaming the texture in.

My use case is to load the mesh to analyze the MeshPtr to manipulate materials.

Code: Select all

OgreMain_d.dll!Ogre::ZipDataStream::read(void * buf=0x00000019b7d6eb00, unsigned __int64 count=128) Line 371
OgreMain_d.dll!Ogre::Image2::load(Ogre::SharedPtr<Ogre::DataStream> & stream={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & type={...}) Line 713
OgreMain_d.dll!Ogre::TextureGpuManager::processLoadRequest(Ogre::ObjCmdBuffer * commandBuffer=0x000001ca8fd93860, Ogre::TextureGpuManager::ThreadData & workerData={...}, const Ogre::TextureGpuManager::LoadRequest & loadRequest={...}) Line 2587
OgreMain_d.dll!Ogre::TextureGpuManager::_updateStreaming() Line 2933
OgreMain_d.dll!Ogre::TextureGpuManager::_updateStreamingWorkerThread(Ogre::ThreadHandle * threadHandle=0x000001ca88dfdfe0) Line 2519
OgreMain_d.dll!Ogre::updateStreamingWorkerThread(Ogre::ThreadHandle * threadHandle=0x000001ca88dfdfe0) Line 2511
OgreMain_d.dll!Ogre::updateStreamingWorkerThread_internal(void * argName=0x000001ca88dfdfe0) Line 81
kernel32.dll!00007ffda5097344()
ntdll.dll!00007ffda67a26b1()

Code: Select all

OgreMain_d.dll!Ogre::Exception::Exception(int num=7, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & desc={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & src={...}, const char * typ=0x00007ffd30ab8888, const char * fil=0x00007ffd30d9e330, long lin=377) Line 62
OgreMain_d.dll!Ogre::InternalErrorException::InternalErrorException(int inNumber=7, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & inDescription={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & inSource={...}, const char * inFile=0x00007ffd30d9e330, long inLine=377) Line 177
OgreMain_d.dll!Ogre::ExceptionFactory::throwException(Ogre::Exception::ExceptionCodes code=ERR_INTERNAL_ERROR, int number=7, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & desc={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & src={...}, const char * file=0x00007ffd30d9e330, long line=377) Line 289
OgreMain_d.dll!Ogre::ZipDataStream::read(void * buf=0x000001fdb333a4f0, unsigned __int64 count=156730) Line 375
OgreMain_d.dll!Ogre::MemoryDataStream::MemoryDataStream(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & name={...}, const Ogre::SharedPtr<Ogre::DataStream> & sourceStream={...}, bool freeOnClose=true, bool readOnly=false) Line 314
OgreMain_d.dll!Ogre::Mesh::prepareImpl() Line 161
OgreMain_d.dll!Ogre::Resource::load(bool background=false) Line 231
OgreMain_d.dll!Ogre::MeshManager::load(const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & filename={...}, const std::basic_string<char,std::char_traits<char>,std::allocator<char> > & groupName={...}, Ogre::BufferType vertexBufferType=BT_IMMUTABLE, Ogre::BufferType indexBufferType=BT_IMMUTABLE, bool vertexBufferShadowed=true, bool indexBufferShadowed=true) Line 127
Sample_V2Mesh.exe!Demo::V2MeshGameState::createScene01() Line 33
Sample_V2Mesh.exe!Demo::BaseSystem::createScene01() Line 29
Sample_V2Mesh.exe!Demo::MainEntryPoints::mainAppSingleThreaded(HINSTANCE__ * hInst=0x00007ff6f20a0000, HINSTANCE__ * hPrevInstance=0x0000000000000000, char * strCmdLine=0x000001fd964548b7, int nCmdShow=10) Line 82
Sample_V2Mesh.exe!WinMainApp(HINSTANCE__ * hInst=0x00007ff6f20a0000, HINSTANCE__ * hPrevInstance=0x0000000000000000, char * strCmdLine=0x000001fd964548b7, int nCmdShow=10) Line 16
Sample_V2Mesh.exe!WinMain(HINSTANCE__ * hInst=0x00007ff6f20a0000, HINSTANCE__ * hInst2=0x0000000000000000, char * strCmdLine=0x000001fd964548b7, int intParam=10) Line 35
Sample_V2Mesh.exe!invoke_main() Line 107
Sample_V2Mesh.exe!__scrt_common_main_seh() Line 288
Sample_V2Mesh.exe!__scrt_common_main() Line 331
Sample_V2Mesh.exe!WinMainCRTStartup() Line 17
kernel32.dll!00007ffda5097344()
ntdll.dll!00007ffda67a26b1()

[dark_sylinc]

Oh.

Then it is our bug.

Can you tell me more about your zip setups so I can easily repro?

[Hilarius86]

Here is the link to my zip
https://cloud.ibtnet.de/s/A9NsywFK7Sf3CEZ
After adding an entry in the resources.cfg and with the sample code, the problem should already reproduce.

If I reduce the resolution down from 4k I can also negate the race condition. Sadly the models are user generated so I can't easily limit that aspect.