STL background ressourceloading threading

[Thyrion]

If i'm right, in the current VS2015 version the only missing part is the std::shared_mutex.
But if a stable VS 2015 update 2 is available it should be usable:
https://blogs.msdn.microsoft.com/vcblog ... -complete/

CTP Release already here:
https://www.microsoft.com/germany/techw ... licht.aspx

Code: Select all

#ifndef __OgreThreadDefinesSTL_H__
#define __OgreThreadDefinesSTL_H__

#include <condition_variable>
#include <mutex>
#include <shared_mutex>
#include <thread>
#include <iostream>
#include <queue>
#include <chrono>

#define OGRE_TOKEN_PASTE(x, y) x ## y
#define OGRE_TOKEN_PASTE_EXTRA(x, y) OGRE_TOKEN_PASTE(x, y)


#define OGRE_LOCK_AUTO_MUTEX std::lock_guard<std::recursive_mutex> ogreAutoMutexLock(OGRE_AUTO_MUTEX_NAME)
#define OGRE_LOCK_MUTEX(name) std::lock_guard<std::recursive_mutex> OGRE_TOKEN_PASTE_EXTRA(ogrenameLock, __LINE__) (name)
#define OGRE_LOCK_MUTEX_NAMED(mutexName, lockName) std::lock_guard<std::recursive_mutex> lockName(mutexName)
#define OGRE_THREAD_SYNCHRONISER(sync) std::condition_variable  sync
#define OGRE_THREAD_SLEEP(ms) std::this_thread::sleep_for(std::chrono::microseconds(ms))

#define OGRE_AUTO_MUTEX mutable std::recursive_mutex OGRE_AUTO_MUTEX_NAME
#define OGRE_MUTEX(name) mutable std::recursive_mutex name
#define OGRE_STATIC_MUTEX(name) static std::recursive_mutex name
#define OGRE_STATIC_MUTEX_INSTANCE(name) std::recursive_mutex name
// like OGRE_AUTO_MUTEX but mutex held by pointer
#define OGRE_AUTO_SHARED_MUTEX mutable std::recursive_mutex *OGRE_AUTO_MUTEX_NAME
#define OGRE_LOCK_AUTO_SHARED_MUTEX assert(OGRE_AUTO_MUTEX_NAME); std::lock_guard<std::recursive_mutex> ogreAutoMutexLock(*OGRE_AUTO_MUTEX_NAME)
#define OGRE_NEW_AUTO_SHARED_MUTEX assert(!OGRE_AUTO_MUTEX_NAME); OGRE_AUTO_MUTEX_NAME = new std::recursive_mutex()
#define OGRE_DELETE_AUTO_SHARED_MUTEX do { assert(OGRE_AUTO_MUTEX_NAME); delete OGRE_AUTO_MUTEX_NAME; } while (0)
#define OGRE_COPY_AUTO_SHARED_MUTEX(from) assert(!OGRE_AUTO_MUTEX_NAME); OGRE_AUTO_MUTEX_NAME = from
#define OGRE_SET_AUTO_SHARED_MUTEX_NULL OGRE_AUTO_MUTEX_NAME = 0
#define OGRE_MUTEX_CONDITIONAL(mutex) if (mutex)
#define OGRE_THREAD_WAIT(sync, mutex, lock) sync.wait(lock)
#define OGRE_THREAD_NOTIFY_ONE(sync) sync.notify_one()
#define OGRE_THREAD_NOTIFY_ALL(sync) sync.notify_all()
// Read-write mutex
#define OGRE_RW_MUTEX(name) mutable std::shared_mutex name
#define OGRE_LOCK_RW_MUTEX_READ(name) std::shared_lock<std::shared_mutex> OGRE_TOKEN_PASTE_EXTRA(ogrenameLock, __LINE__) (name)
#define OGRE_LOCK_RW_MUTEX_WRITE(name) std::unique_lock<std::shared_mutex> OGRE_TOKEN_PASTE_EXTRA(ogrenameLock, __LINE__) (name)
// Thread-local pointer
#define OGRE_THREAD_POINTER(T, var) std::thread_specific_ptr<T> var
#define OGRE_THREAD_POINTER_INIT(var) var(&deletePtr)
#define OGRE_THREAD_POINTER_VAR(T, var) std::thread_specific_ptr<T> var (&deletePtr<T>)
#define OGRE_THREAD_POINTER_SET(var, expr) var.reset(expr)
#define OGRE_THREAD_POINTER_GET(var) var.get()
#define OGRE_THREAD_POINTER_DELETE(var) var.reset(0)
// Thread objects and related functions
#define OGRE_THREAD_TYPE std::thread
#define OGRE_THREAD_CREATE(name, worker) std::thread* name = OGRE_NEW_T(std::thread, MEMCATEGORY_GENERAL)(worker)
#define OGRE_THREAD_DESTROY(name) OGRE_DELETE_T(name, thread, MEMCATEGORY_GENERAL)
#define OGRE_THREAD_HARDWARE_CONCURRENCY std::thread::hardware_concurrency()
#define OGRE_THREAD_CURRENT_ID std::this_thread::get_id()
#define OGRE_THREAD_WORKER_INHERIT
// Utility
#define OGRE_THREAD_ID_TYPE std::thread::id
#define OGRE_THREAD_YIELD std::this_thread::yield()

#endif

right?

[Thyrion]

hmm there's still this one to replace:

Code: Select all

#define OGRE_THREAD_POINTER(T, var) boost::thread_specific_ptr<T> var

any idea?
the following doesn't work ...

Code: Select all

thread_local  <std::shared_ptr<T>>  var

or

Code: Select all

thread_local <T>  var

[c6burns]

AFAIK you'd have to implement that yourself to get portability. It won't be rocket science, there's just 2 APIs to wrap: POSIX and WIN32. I'm sure if you crack open boost's implementation it will be a single compilation unit/header, and you will see that's exactly what they've done. Just steal theirs

[Thyrion]

I wanted to try out, if i get the threading work with standard librarys.
Got some weird crashes in the stl thread files, where i dont want to dig deeper for now and gave up.

I'm using boost for threading now (ogre 2.1).
But it seems the rendering still freezes if a new item enters the camera view...

Code: Select all

#define OGRE_THREAD_SUPPORT 1
#define OGRE_THREAD_PROVIDER 1

But no one mentioned problems with the backgroundloader, so maybe i missed someting?
In the current state the background loading seems to be useless.

[Thyrion]

The shader compilation seems to freeze the renderer.

What's the best practice?

Should that not be made in the background loading process?
Or should the shader compilation for all needed materials be done before loading a scene?
Can we precompile the shaders and load them at runtime?

[dark_sylinc]

Or should the shader compilation for all needed materials be done before loading a scene?
Can we precompile the shaders and load them at runtime?

See the Microcode cache post.

Note that on D3D11 the cache is system-agnostic but in GL the cache only works for that particular system until the drivers are updated.
The advice I give is to render a few frames with all/most possible material combinations you'll have on the real scene in order to warm up our caches. You could even run a scene after installation, fill all caches, and save it to disk to avoid delays during the actual game.

If the game later discovers the cache is no longer valid (i.e. GL drivers have been updated) then rebuild the cache again, just once (and inform the player in a clever way, e.g. "System changes detected. Optimizing performance cache. Please wait...").

[Transporter]

Replacing boost/TBB threading with C++11 is not an option, because C++11 is not available in all compilers! There are a lot of people which are still using VS2010, VS2012 or VS2013 and not VS2015.

As c6burns has noted, a wrapper for POSIX and WIN32 is required. An additional option could be C++11 threads in this module.

[dark_sylinc]

Forums are great for communicating with people, but unfortunately they suck at making something visible for everyone, so I'm going to write this again:

• STL's threading has no place in a high performance graphics library.
• In particular, Visual Studio's STL implementation of threads sucks.
• If it were by me, the current background resource loading mechanism could die in a fire. It's only been kept there because there is no replacement at the moment and some users have successfully been able to use it. But I don't particularly recommend it. But if you can make it work for you, and you're happy, I won't stand in your way.

[hydexon]

I've used background resource loading system before, and it sucks in my opinion, i have to do nasty multithreaded coding in order to avoid race conditions and deadlocks (probably i misused it) in my own code, now i'm using a very different approach without relying exclusively from the OGRE's resource system, using the Helium's Project Asset System, it's a bit more complex than OGRE's but the background loading is pretty good for me.

I felt so well since dark_sylinc told us OGRE 2.1 multithreading doesn't need Boost/TBB/Poco because those libraries are a PITA to build (specially TBB and Boost), i prefer the old background loading system die, and if i can decouple OGRE and the ResourceManager system to access data.

[Thyrion]

drop the crap