[GSoC 2013 - accepted] Ogre 2.0

Threads related to Google Summer of Code
Post Reply
drwbns
Orc Shaman
Posts: 788
Joined: Mon Jan 18, 2010 6:06 pm
Location: Costa Mesa, California
x 24

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by drwbns »

For clarification, are you planning on completely omitting boost and other threading options from your version or something similar to what klaim said?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by dark_sylinc »

Klaim wrote:Just to clarify, here you are talking exclusively about rendering/updating animations/particles, not resource loading, right?
Yes
Klaim wrote: If you use custom threads, then I see a potential oversubscription problems in the coming future (or with my current code).
You must tell how many threads you want Ogre to create when creating the scene manager. That is in your control.
Klaim wrote: I don't agree because I have no confidence (by experience) in custom code doing low level synchronization. It's not in you I have no confidence, it's in low level synchronization code.
I guess it's ok for now if you don't find any problem, but I also suspect library code to be of higher quality and certainly both easier to maintain and debug
than anything custom we can put Ogre code. Just for the sake of keeping Ogre only about graphics, that's not a good idea.
I understand that adding libraries right now is not really ideal, but I suspect there is enough constructs in C++11 standard library to easily
write the same semantic. I suggest using the standard library instead (for long-term), NOT NOW, but as soon as the OGRE version is stable (somewhere next year?)
These things frankly scares me a lot (all my current code is concurrent).
Why everyone puts so much of a fuss on threading libraries?
I'm going to sound arrogant on this one: It feels that I'm trying to do 2+2 and everyone is suggesting I should use a bignum library.
A thread barrier is one of the most basic synchronization primitives. We could even use OpenMP if it weren't for poor compiler support.

I've done a lot of very hard multithreading programming, and this job is the most simple threading example I can think of; usually done best using native synchronization primitives.
It's synchronous multithreading, not asynchronous. Threading libraries are meant to deal with async multithreading problems. For synchronous multithreading, all libraries just wrap up the basic synchronization primitives for a given platform and call it a day.
drwbns wrote:For clarification, are you planning on completely omitting boost and other threading options from your version or something similar to what klaim said?
Omitting boost and other threading options. In PThreads terms, all I'll be using is a pthread_barrier_t and pthread_create (and pthread_join on shutdown).
On Windows terms, two Semaphores + an interlocked increment to emulate a barrier, which is the standard way of emulating them (Sync Barriers haven't been introduced until Windows 8... yeah no comments), CreateThread, and WaitForMultipleObjects.
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Klaim »

dark_sylinc wrote:
Klaim wrote: If you use custom threads, then I see a potential oversubscription problems in the coming future (or with my current code).
You must tell how many threads you want Ogre to create when creating the scene manager. That is in your control.
Perfect.
Why everyone puts so much of a fuss on threading libraries?
Because both on the correctness, performance and maintenance side, their implementation is a moving target at this time. Their interface is not.
I'm going to sound arrogant on this one: It feels that I'm trying to do 2+2 and everyone is suggesting I should use a bignum library.
A thread barrier is one of the most basic synchronization primitives. We could even use OpenMP if it weren't for poor compiler support.

I've done a lot of very hard multithreading programming, and this job is the most simple threading example I can think of; usually done best using native synchronization primitives.
Again I know you know what you're doing on the implementation level. What I'm saying is that this is not a good long-term strategy, in the same way choosing standard library container was a good long-term strategy for Ogre, in general.
It's synchronous multithreading, not asynchronous. Threading libraries are meant to deal with async multithreading problems. For synchronous multithreading, all libraries just wrap up the basic synchronization primitives for a given platform and call it a day.
Yes exactly. That's the point actually. Even std::future::get() currently is considered synchronous multithreading (with simple and readable interface).

Now again, I just think - on the long term - a library solution will be a better bet than specific PThread code (in particular on Windows and derived), so it's not important "now" but I couldn't avoid reacting to this. Feel free to ignore my comments for now. :)
Owen
Google Summer of Code Student
Google Summer of Code Student
Posts: 91
Joined: Mon May 01, 2006 11:36 am
x 21

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Owen »

The C++ standard library doesn't have thread barriers, either now or in C++14, so its' a completely irrelevant distraction.

Wrapping the OS threading primitives is neither difficult or tricky. Implementing a barrier on Windows is somewhat tricky but nothing unreasonable.

To be honest, I find it kind of ridiculous that for years Ogre has been depending on something as heavyweight as Boost for something as trivial as a mutex; meanwhile implementing its' own atomics and shared_ptr (<-- the latter was, for a long time, implemented particularly badly BTW). A thread barrier implementation is really no more difficult than implementing an efficient-as-possible shared_ptr (as Ogre::SharedPtr is now)

If somebody really wants it, they can write an implementation which delegates the work to boost.thread or similar.

----

dark_sylinc: I quite like the value-type semantics and general API of boost::thread::barrier. What do you say to matching it?
drwbns
Orc Shaman
Posts: 788
Joined: Mon Jan 18, 2010 6:06 pm
Location: Costa Mesa, California
x 24

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by drwbns »

I dunno, to me it sounds as though dark_sylinc wants to use a one off threading choice for this one without hassling with the overweight boost library. Which is what many have complained about before. If it's as simple as he says it is, there's no reason why other threading parts of Ogre couldn't use a lightweight choice as well. I don't see this as being a long-term problem when performance and portability outweigh so heavily.
PhilipLB
Google Summer of Code Student
Google Summer of Code Student
Posts: 550
Joined: Thu Jun 04, 2009 5:07 pm
Location: Berlin
x 108

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by PhilipLB »

And if it's nicely abstracted, there is no obstacle to overwrite the Ogre-solution with Boost. And it would be possible to later exchange the implementation with Boost without braking the API in a later Ogre version if needed. -> Sounds good to me.
Google Summer of Code 2012 Student
Topic: "Volume Rendering with LOD aimed at terrain"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: Mattan Furst


Volume GFX, accepting donations.
User avatar
Zonder
Ogre Magi
Posts: 1168
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 73

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Zonder »

I also think just using a light weight solution is fine no need to bloat the library. Once it's working across platforms it's not like it's something that will need a lot of maintenance.
There are 10 types of people in the world: Those who understand binary, and those who don't...
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Klaim »

PhilipLB wrote:And if it's nicely abstracted, there is no obstacle to overwrite the Ogre-solution with Boost. And it would be possible to later exchange the implementation with Boost without braking the API in a later Ogre version if needed. -> Sounds good to me.
I agree, as long as it's abstracted it's ok for me too.
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by spacegaier »

PhilipLB wrote:And if it's nicely abstracted, there is no obstacle to overwrite the Ogre-solution with Boost. And it would be possible to later exchange the implementation with Boost without braking the API in a later Ogre version if needed. -> Sounds good to me.
+1
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
al2950
OGRE Expert User
OGRE Expert User
Posts: 1227
Joined: Thu Dec 11, 2008 7:56 pm
Location: Bristol, UK
x 157

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by al2950 »

PhilipLB wrote:And if it's nicely abstracted, there is no obstacle to overwrite the Ogre-solution with Boost. And it would be possible to later exchange the implementation with Boost without braking the API in a later Ogre version if needed. -> Sounds good to me.
+2!
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by spacegaier »

Another interesting blog post my Mathias.

http://yosoygames.com.ar/wp/2013/09/shadow-mapping-oops
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by dark_sylinc »

Uploaded the Ogre 2.0 Porting Manual (DRAFT)

Links are also on the first page. Enjoy!
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Klaim »

Good work (again)!

Some feedbacks/questions on this document:

0. Globally, almost everything is clear, well done. I like that there are images. The only part that I didn't quite understand is ...
1. ... "2.2 How to debug MovableObjects' (and Nodes) data": I'm lost starting from "Nevertheless, debugging SSE builds directly isn't that difficult.". I'm not sure if it's because I'm not an expert on the subject or it's the way you explain it. I get a glimpse of understanding after a lot of re-reading. Am I correct that you're saying SSE builds imply different arrays for different sizes of vector values, then mIndex tells you which array is to be used? Or something like that...
2. "2.2.1 Interpreting ArrayVector3": am I correct that m_chunkBase (in the image) is a transform matrix? Because it was not immediately clear to me (but maybe because I'm not used to look at them directly in the eye). If I'm correct, saying it directly (maybe in the variable name?) would help interpreting the data when debugging.
3. As suggested on Twitter, if there are changes in relationships between types in the whole Ogre organization, it would be nice (as a clarification) if there would be a section with a diagram before and a diagram after the changes. Maybe the before one would be the one in the manual.
4. Could you add a quick clarification (just after "names are optional" maybe) about the SceneMemoryMgrTypes argument in the entity creation function? So that the change on the way to managing static/dynamic is clear. As you say later "don't: Create objects that never move, rotate or scale as SCENE_DYNAMIC" without saying what's it's about, I think it's worth having a short section explaining it. Maybe just copy/pasting what you already wrote in this discussion.
5. About threading, did I understand correctly that threads are associated to scene managers, so if you have several scenes to be rendered in the same time you need to keep in mind that each scene will have it's own set of threads (or not)?
6. Am I correct that if numThreads (in the scene/thread initialization example) is 0, then there is just no additional worker thread created?
7. The parts explaining what is threaded, what can be done concurrently or not are definitely useful, very good!
8. "At the time of writing only HW Basic supports passing the custom parameters. All other techniques will ignore
it." Did you mean it's not possible or it's not implemented? (I'm not a specialist in Instancing nor in shaders-related subjects in general, so I have no idea)
9. Is CompositorManager2 a final name? Is it supposed to replace CompositorManager later? Or did you mean it's an alternative way of doing it and you think the older way should be kept?
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by spacegaier »

Haven't read the whole guide yet, but on the very last page in the "Don't" section: Are the first two points correct?

In the first one, you mention and STATIC_SCENE and SCENE_STATIC? Is that a typo?
In the second one, you say objects that never move should be in SCENE_DYNAMIC? Shouldn't it be the other way around? Or maybe one has to combine those statements with the "Don't" header to get the right meaning. In that case, please change the don't points to already include that "don't". As an example: The last bullet point would become: "Do not specify more threads than the number of available cores."
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by dark_sylinc »

Thanks for the feedback, it helps me know what is clear and what's not and what I should I improve.
Klaim wrote: 1. ... "2.2 How to debug MovableObjects' (and Nodes) data": I'm lost starting from "Nevertheless, debugging SSE builds directly isn't that difficult.". I'm not sure if it's because I'm not an expert on the subject or it's the way you explain it. I get a glimpse of understanding after a lot of re-reading. Am I correct that you're saying SSE builds imply different arrays for different sizes of vector values, then mIndex tells you which array is to be used? Or something like that...
Yes. When SSE2 is used, ARRAY_PACKED_REALS = 4; and mOwner is declared as:

Code: Select all

Node **mOwner[ARRAY_PACKED_REALS]
So In other words, mOwner[4]; in C compatibility version (non-SIMD builds), ARRAY_PACKED_REALS = 1; hence mOwner[1]
To know which one is yours, read mOwnder[mIndex];
Klaim wrote: 2. "2.2.1 Interpreting ArrayVector3": am I correct that m_chunkBase (in the image) is a transform matrix? Because it was not immediately clear to me (but maybe because I'm not used to look at them directly in the eye). If I'm correct, saying it directly (maybe in the variable name?) would help interpreting the data when debugging.
No. ArrayVector3 packs 4 Vector3 together (in an SSE2 build, because ARRAY_PACKED_REALS = 4)
So you're looking at a class that contains 4 vectors in one. If the vectors were named A, B, C, D; the picture says:
m_chunkBase[0] = { D.x, C.x, B.x, A.x }
m_chunkBase[1] = { D.y, C.y, B.y, A.y }
m_chunkBase[2] = { D.z, C.z, B.z, A.z }

Klaim wrote: 4. Could you add a quick clarification (just after "names are optional" maybe) about the SceneMemoryMgrTypes argument in the entity creation function? So that the change on the way to managing static/dynamic is clear. As you say later "don't: Create objects that never move, rotate or scale as SCENE_DYNAMIC" without saying what's it's about, I think it's worth having a short section explaining it. Maybe just copy/pasting what you already wrote in this discussion.
mmmm.... ok, I'll re-read to understand what's not clear.
Klaim wrote: 5. About threading, did I understand correctly that threads are associated to scene managers, so if you have several scenes to be rendered in the same time you need to keep in mind that each scene will have it's own set of threads (or not)?
Yes.
Klaim wrote: 6. Am I correct that if numThreads (in the scene/thread initialization example) is 0, then there is just no additional worker thread created?
Yes, though I should clarify that there should be one worker thread (numThreads must be > 0). One CPU = One worker thread.
Don't worry, Ogre will throw you telling that numThreads must be greater than 0.

Klaim wrote: 8. "At the time of writing only HW Basic supports passing the custom parameters. All other techniques will ignore
it." Did you mean it's not possible or it's not implemented? (I'm not a specialist in Instancing nor in shaders-related subjects in general, so I have no idea)
Ooops, sorry. Implementing it in HW VTF is possible (though I have a couple of internal disagreements with myself on how to implement it: put it in the VTF or pass it as extra texcoords).
As for regular VTF and ShaderBased, it's not impossible to implement but highly inadvisable (it would hurt the gpu very bad)
Klaim wrote: 9. Is CompositorManager2 a final name? Is it supposed to replace CompositorManager later? Or did you mean it's an alternative way of doing it and you think the older way should be kept?
**Sigh** There was a name clashing with the old CompositorManager. Now it is removed, but at the time the name clashing was there. In other words CompositorManager2 completely replaces CompositorManager.
spacegaier wrote:Haven't read the whole guide yet, but on the very last page in the "Don't" section: Are the first two points correct?

In the first one, you mention and STATIC_SCENE and SCENE_STATIC? Is that a typo?
Ooops, I you see STATIC_SCENE, I mean SCENE_STATIC (it's a typo)
spacegaier wrote: In the second one, you say objects that never move should be in SCENE_DYNAMIC? Shouldn't it be the other way around? Or maybe one has to combine those statements with the "Don't" header to get the right meaning. In that case, please change the don't points to already include that "don't". As an example: The last bullet point would become: "Do not specify more threads than the number of available cores."
It's in the "Don't" section. You're supposed to read it as "Don't create objects that never move, rotate or scale as SCENE_DYNAMIC"
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by spacegaier »

dark_sylinc wrote:
spacegaier wrote: In the second one, you say objects that never move should be in SCENE_DYNAMIC? Shouldn't it be the other way around? Or maybe one has to combine those statements with the "Don't" header to get the right meaning. In that case, please change the don't points to already include that "don't". As an example: The last bullet point would become: "Do not specify more threads than the number of available cores."
It's in the "Don't" section. You're supposed to read it as "Don't create objects that never move, rotate or scale as SCENE_DYNAMIC"
I almost assumed so, but it is highly misleading, so as I wrote earlier, I would urge you to rephrase that so that all statements in the "Don't" section also include the "do not" part within them. Otherwise people will get confused!

As an example: The last bullet point would become: "Do not specify more threads than the number of available cores." Much clearer.
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by dark_sylinc »

The manual has been updated with some of the suggestions made here, and fixed lots of grammar mistakes.
Download it again.
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Klaim »

dark_sylinc wrote:
Klaim wrote: 2. "2.2.1 Interpreting ArrayVector3": am I correct that m_chunkBase (in the image) is a transform matrix? Because it was not immediately clear to me (but maybe because I'm not used to look at them directly in the eye). If I'm correct, saying it directly (maybe in the variable name?) would help interpreting the data when debugging.
No. ArrayVector3 packs 4 Vector3 together (in an SSE2 build, because ARRAY_PACKED_REALS = 4)
So you're looking at a class that contains 4 vectors in one. If the vectors were named A, B, C, D; the picture says:
m_chunkBase[0] = { D.x, C.x, B.x, A.x }
m_chunkBase[1] = { D.y, C.y, B.y, A.y }
m_chunkBase[2] = { D.z, C.z, B.z, A.z }
Ok then what I don't understand is either
1. if B is this object's position, what are C and D? Other's node's vectors? A is used for management if I understand correctly.
2. how to you know which column corresponds to the current object's position vector?

Sorry, I'm a bit confused about this.
I just quickly read the new version but I still miss that information.
Klaim wrote: 4. Could you add a quick clarification (just after "names are optional" maybe) about the SceneMemoryMgrTypes argument in the entity creation function? So that the change on the way to managing static/dynamic is clear. As you say later "don't: Create objects that never move, rotate or scale as SCENE_DYNAMIC" without saying what's it's about, I think it's worth having a short section explaining it. Maybe just copy/pasting what you already wrote in this discussion.
mmmm.... ok, I'll re-read to understand what's not clear.
What I meant exactly is that in this specific document you never explain the meaning of SCENE_DYNAMIC and SCENE_STATIC but you mention these values, while it's a new thing replacing an previous thing. So I think a short explanation of this in the beginning would be helpful. I know what these are because I followed this thread, it's just the document missing the info.
Klaim wrote: 6. Am I correct that if numThreads (in the scene/thread initialization example) is 0, then there is just no additional worker thread created?
Yes, though I should clarify that there should be one worker thread (numThreads must be > 0). One CPU = One worker thread.
Don't worry, Ogre will throw you telling that numThreads must be greater than 0.
Let me rephrase the question: if I want Ogre to never spawn any other thread, just use the thread calling renderOneFrame() to do everything, should I set 1? Or do you mean there is always at least one thread that will be associated to each scene?
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by dark_sylinc »

Klaim wrote: What I meant exactly is that in this specific document you never explain the meaning of SCENE_DYNAMIC and SCENE_STATIC but you mention these values, while it's a new thing replacing an previous thing. So I think a short explanation of this in the beginning would be helpful. I know what these are because I followed this thread, it's just the document missing the info.
OMG I totally forgot that I didn't write about that and thought I did!

I apologize for that. The manual has been updated again. There's new section devoted to that.
Last edited by dark_sylinc on Sat Sep 28, 2013 7:31 pm, edited 1 time in total.
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Klaim »

dark_sylinc wrote: 1. No Node is used for "management". C and D are other node's vectors like you said. A is also another node's position.
2. By looking at the value of mIndex. If mIndex = 0; then I should look at the 4th coloumn. If mIndex = 1 I should look at the 3rth column, if mIndex = 3 I should be looking at the 1st column.
Ooookaaaayyy now I got it! From reading I thought that the mIndex was the index of the parent only, not of all the related data.
Maybe because the section "2.2.1 Interpreting ArrayVector3" never mention mIndex (you insist a lot of it before but the section made a separation in my mind).

Ok I think I understand why that section was confusing, it might be because of the examples themselves. What I mean is:
Here the debugger is telling us that the center of the Aabb in local space is at (100; 100; 100)
might be improved by mentionning why we are taking this value and not another (yes it's repeatition but it's necessary in human readable doc ;_; ):
Here the debugger is telling us that the center of the Aabb in local space is at (100; 100; 100), which is the vector made of the values at index mIndex in reverse order.
Or something similar.
Then:
Note that if, for example, the 4th element (in this case it reads (0, 0, 0)) is an empty slot (i.e. there are only 3
entities in the entire scene); it could contain complete garbage; even NaNs. This is ok. We fill the memory
with valid values after releasing memory slots to prevent NaNs & INFs; as some architectures are slowed
down when such floating point special is present; but it is possible that some of them slip through (or it is also
possible a neighbour Entity actually contains an infinite extent, for example).
The image shows (0,0,0) for both the 4th and the 0th element so I mixed both. It's a bit confusing, but I'm not sure how to improve this part.

Klaim wrote: Let me rephrase the question: if I want Ogre to never spawn any other thread, just use the thread calling renderOneFrame() to do everything, should I set 1? Or do you mean there is always at least one thread that will be associated to each scene?
Ogre always has to spawn another another thread.
But you don't have to worry that these two threads (Ogre main thread and its worker thread) run concurrently because they won't. When Ogre needs the worker thread to run, the main thread will pause.

And yes, there is some unnecessary OS overhead (thread context switching) for doing that, which I was worried about but when I benchmarked the old vs new code, performance drop was less than 1% for single core (which could easily be attributed to noise); and the amount of effort needed to maintain a single threaded path and a multithreaded path was too big for such minimal margin.
I didn't question the threading impact yet because I want to focus on helping you with clarifying what is intended (and happening) in the current setup.
Improving on this front can be done later I guess.

If I know exactly what's the cost of each scene, at least I can do some prevention (even if that setup will be a bit expensive for me I think, but I'll check and report after trying Ogre 2.0 ).
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by dark_sylinc »

Klaim wrote: If I know exactly what's the cost of each scene, at least I can do some prevention (even if that setup will be a bit expensive for me I think, but I'll check and report after trying Ogre 2.0 ).
Though it may be worth saying that we're aiming towards updating the SceneManager through the CompositorManager, and SceneManagers would be updated sequentially. I don't even know if that can be done in parallel.
Since they're updated sequentially, you should provide numThreads with the total number of cores, regardless of how many SceneManagers you create.

Yes, I know, that can spawn a lot of threads (3 SceneManagers in a 6-core machine = 18 threads) unnecessarily. Perhaps we can later revise this (not now, there's a lot to do more important) so that they can share worker threads as long as you promise not to update these managers at the same time.
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Klaim »

dark_sylinc wrote:
Klaim wrote: If I know exactly what's the cost of each scene, at least I can do some prevention (even if that setup will be a bit expensive for me I think, but I'll check and report after trying Ogre 2.0 ).
Though it may be worth saying that we're aiming towards updating the SceneManager through the CompositorManager, and SceneManagers would be updated sequentially. I don't even know if that can be done in parallel.
In my specific case, I will need to sometime prepare a scene before it's visible, so I'll need to create a scene, once it's filled with objects I'll begin rendering in a texture, and at some point
I'll end up keeping that new scene and releasing the old one (not sure when though).
I also have a thread pool (TBB based) that I use for almost anything other than Ogre manipulations.
That's why I'm saying creating a scene will be costly for me because all that have to happen without rendering stopping (filling the new scene will happen progressively in time).
My goal is to have no loading screen, but having a loading that happen while I'm playing is ok (for example like corridors in games like Metroid).
Since they're updated sequentially, you should provide numThreads with the total number of cores, regardless of how many SceneManagers you create.

Yes, I know, that can spawn a lot of threads (3 SceneManagers in a 6-core machine = 18 threads) unnecessarily. Perhaps we can later revise this (not now, there's a lot to do more important) so that they can share worker threads as long as you promise not to update these managers at the same time.
Indeed.

As a subject for a future discussion, it makes me think a lot about the Executor idea proposed by Google and Microsoft to the C++ standard: http://www.open-std.org/JTC1/SC22/WG21/ ... /n3731.pdf
Apparently it's the most flexible way they could organize their different setup.

That is, if there is a simple virtual class to provide to Ogre which guarantees to execute tasks as soon as possible, then you'll "just" have to have parallel_for_each and maybe a map_reduce algorithms built over that, which then allows easily sharing a pool of worker threads with several libraries (or having a specific one for each one for example).

Anyway I'm pointing to this in case it can solve the potential systematic oversubscription when using TBB for example.

Note that for the moment I don't get yet the performance slowdown of having 1 more worker thread than the number of cores (Ogre having it's own thread in my current setup) because I'm not finished implementing features so it's not stressed yet on my machine (and my tester's). So I don't mind for now personally, but I indeed see the potential future problem.
User avatar
Mako_energy
Greenskin
Posts: 125
Joined: Mon Feb 22, 2010 7:48 pm
x 9

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Mako_energy »

dark_sylinc,

I have what may be an unusual (or ignorant) gripe regarding the attachment and visibility changes made to movable objects. I think forcing objects to always be in the scene may get in the way of some operations in some setups. Such as when you want to prepare a scene in advance so you don't need to create objects during gameplay. I understand that there isn't much difference between toggling visibility and removing from the scene, but one critical difference is what shows up in ray/spacial scene queries that devs may want to perform.

It is also counter-intuitive to have an accessible setting (set/get Visible) be clobbered based on it's attached state, especially when coming from a system that made a distinction between in-scene and visible. Personally I value that distinction, but if there is a technical reason that cannot be, I will trust your judgement. I hope I articulated this well enough, I feel it's not as clear as it could be.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5296
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by dark_sylinc »

Mako_energy wrote:I have what may be an unusual (or ignorant) gripe regarding the attachment and visibility changes made to movable objects. I think forcing objects to always be in the scene may get in the way of some operations in some setups. Such as when you want to prepare a scene in advance so you don't need to create objects during gameplay. I understand that there isn't much difference between toggling visibility and removing from the scene, but one critical difference is what shows up in ray/spacial scene queries that devs may want to perform.

It is also counter-intuitive to have an accessible setting (set/get Visible) be clobbered based on it's attached state, especially when coming from a system that made a distinction between in-scene and visible. Personally I value that distinction, but if there is a technical reason that cannot be, I will trust your judgement. I hope I articulated this well enough, I feel it's not as clear as it could be.
Actually that was one of the few little-big details that came up organically during development. I didn't plan that up, nor foresee it.

The way the design was going, it started working into that direction. It just naturally "felt right". That, and a very technical reason that there were a lot of memory pointers the hardware prefetcher was tracking and adding an additional pointer saturated the bandwidth even more and started confusing the prefetcher.
So, when I started to see how to overcome the technical limitation, I realized the new system just worked differently: All objects "are in scene" (well, that notion doesn't exist anymore) and Nodes are just the container of their position in the world that can be parented to other nodes.

By the way, you made me remember that the raycasting system is completely broken and needs to be added to the TODO list. I didn't have time to look into that (I'm sorry!)
Mako_energy wrote: I understand that there isn't much difference between toggling visibility and removing from the scene, but one critical difference is what shows up in ray/spacial scene queries that devs may want to perform.
I cannot stress enough the importance of Visibility Flags. If you want to have objects that are invisible during render but are part of the scene (aka ghosts/phantoms) and as such should show up in ray/spatial scene queries, make the objects visible but put them in a special layer that no viewport will use. Hence ray casts will see it but the objects won't be rendered.

There's a lot of problems that can be easily solved by playing with visibility flags, which people forget about them and start doing complex modifications to the code to get their specific job done and then this code they wrote implementing a "new feature" gets integrated into the main branch.

I know it's a mentality to change, but with a bit of time you'll see the new system is not just faster, but more flexible and easier to use.
User avatar
Mako_energy
Greenskin
Posts: 125
Joined: Mon Feb 22, 2010 7:48 pm
x 9

Re: [GSoC 2013 - accepted] Ogre 2.0

Post by Mako_energy »

dark_sylinc wrote:By the way, you made me remember that the raycasting system is completely broken and needs to be added to the TODO list. I didn't have time to look into that (I'm sorry!)
Hurray! I'm useful!
dark_sylinc wrote:I cannot stress enough the importance of Visibility Flags. If you want to have objects that are invisible during render but are part of the scene (aka ghosts/phantoms) and as such should show up in ray/spatial scene queries, make the objects visible but put them in a special layer that no viewport will use. Hence ray casts will see it but the objects won't be rendered.
Sounds good, but what about if I want objects to be removed from all interactions with the scene? Do you plan to have the updated ray scene query skip something if it isn't visible?
dark_sylinc wrote:There's a lot of problems that can be easily solved by playing with visibility flags, which people forget about them and start doing complex modifications to the code to get their specific job done and then this code they wrote implementing a "new feature" gets integrated into the main branch.
I cannot and will not defend all the different "features" that have appeared over time because of that, but sometimes it does say something about the ease of use for those features if someone is willing to work around the existing way the system is presented. It may be a petty gripe but between tracking a bunch of bitmasks in different places and calling "detach" and knowing I've covered my bases...I'd prefer the latter (which is how it works in 1.x as far as I can tell).
dark_sylinc wrote:I know it's a mentality to change, but with a bit of time you'll see the new system is not just faster, but more flexible and easier to use.
No contest on the first two there, but the third...I hope so.

Like I said before, there is a technical reason for this change so I will trust your choice. But going forward...if there was some API that was able to remove the object from all interactions with the scene in one convenient place (with little to no performance penalty, of course) that would be helpful for me at least. Regardless, I plan to adopt Ogre 2.0 as soon as particles are functional again. And thanks for all the hard work you have put into this.

Edit: I have carefully reviewed the visibility and query flags on the movable object, and assuming queries and rendering is the only interactions with the scene these objects have that could be considered "in world", then I should be able to put together an adequate solution. Still wouldn't mind something easier, but this should be fine.
Post Reply