The roadmap to 1.9 and 2.0

Discussion area about developing with Ogre-Next (2.1, 2.2 and beyond)


User avatar
Zonder
Ogre Magi
Posts: 1168
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 73

Re: The roadmap to 1.9 and 2.0

Post by Zonder »

Can we not come up with a very simple interface for a thread and a task system?

The team can then use a particular library for ogre as standard (something nice and small hopefully ;)). It does allow the community to offer solutions for others as well then like boost or their own implementations like the paradise engines micro threading
There are 10 types of people in the world: Those who understand binary, and those who don't...
User avatar
Xavyiy
OGRE Expert User
OGRE Expert User
Posts: 847
Joined: Tue Apr 12, 2005 2:35 pm
Location: Albacete - Spain
x 87

Re: The roadmap to 1.9 and 2.0

Post by Xavyiy »

Hi guys!

It's really exciting to see you talking about all these ideas, I was looking forward that day since a long time ago :)
(Unfortunately I've been too busy with the university these last two months for participating in the topic, let's hope it's not too late)

1.Finish our new-generation render systems.
Great, IMHO that must be a top priority task, even more than SM role decoupling and general Ogre modularization. We can wait to the end of the summer to see robert_sasu(http://ogre3d.org/forums/viewtopic.php?f=13&t=69655) progress but I think preparing Ogre to be next-gen rendersystems(DX11, OGL3+, ...) ready will need some more work than just one GSoC slot, specially since some features are going to be only compatible with these next-gen rendersystems and that will need a robust API interface.

2.Improved support for current mobile platforms
Great, we must take account that a lot of indie developers are now targeting to mobile platforms more than to PC.

3.State cache for the rendersystems
Fine.

4.Clean up Ogre core and possibly move features to separate components/plugins & 5.SceneManager redesign.
Interesting reads:
By dark_sylinc: http://ogre3d.org/forums/viewtopic.php? ... 50#p453778
By tuan kuranes: http://ogre3d.org/forums/viewtopic.php? ... 50#p453845
IMHO we should first focus the work on the cache-misses related stuff, since that's something that can really improve Ogre performance and at the end it's not that big task.

Also, I want to talk a little about the task-oriented threading system and decoupling the math-related classes from Ogre:
- Threading system:
From my experience in the Paradise Engine (I'm using a custom microthreading library, Paradise Threading), it's worth to write a custom abstraction layer rather than using a 3rd party lib, and this fact take even more importance in Ogre since Intel TBB GPL-named license is not welcome in the majority of legal departments.
In my case, the Paradise Threading has taken me about 2 months coding ~2 hours per day, and it has far more features than the needed in Ogre (it supports data dependencies between tasks, some dynamic-grain-adaptative calculators, etc). Also I had been testing it during these two months implementing a little microthreaded app implementing this paper: http://www.mpi-inf.mpg.de/~ritschel/Pap ... lGlare.pdf (Microthreaded FFTs, etc). A little analisys: http://www.paradise-studios.net/2010/03 ... ironments/

- Math-related classes:
In the Paradise Engine I'm using a custom math templated library. It's quite powerfull and something like it can be a good option in Ogre. BUT I'm not a big fan of decoupling it from OgreMain.

Btw, I'm wondering what's your idea: do you plan just to separate the math-related classes in another lib (something like OgreMath) and maintain all classes in the Ogre namespace(this way you'll be not breaking any existing code, otherwise I don't want to imagine the pain of rewriting every line which uses Ogre::Real, Ogre::Vector3, ...) or adding another namespace?

The advantages of decoupling the Math-related classes is being able to use it withouth having to use the whole OgreMain, but if we mantain the namespace... it'll be a little annoying to use the Ogre namespace in every files or directly Ogre::MathClass.

Personally I don't see any real advantage FOR OGRE in decoupling it into a new dependency (if someone needs the math part, it's just a matter of removing all unneeded files, which can be done in less than 10min, or using any other c++ math solution). Also, all Ogre-related projects will need that dependency...
So, for me, it's not a priority and I naturally think about it like an OgreMain part, not something that needs to be in a different lib.

Xavier
Last edited by Xavyiy on Tue Jun 12, 2012 10:40 pm, edited 1 time in total.
User avatar
Kojack
OGRE Moderator
OGRE Moderator
Posts: 7157
Joined: Sun Jan 25, 2004 7:35 am
Location: Brisbane, Australia
x 534

Re: The roadmap to 1.9 and 2.0

Post by Kojack »

For most uses it would be simple to just copy the needed maths files (there's not much there) and include them directly in another project (this is actually how I use Lua, I never build it as a lib, I just include the files in my project directly. Less hassle). The main problems are that the math classes all use _OgreExport so they think they are a DLL, and they all include ogrestableheaders.h which then includes movableobjects and zip files and resources and other stuff that's definitely not needed. Fix those two things and the math code is easy to insert into another project or build as a lib.

A simple static lib would be ok, but this stuff is pretty small. Since it's not really optional in ogre itself, just making it easily usable in other things should be enough.
(I use my own math lib instead of ogre's for everything non graphical anyway, so I have no personal use for it)
User avatar
Xavyiy
OGRE Expert User
OGRE Expert User
Posts: 847
Joined: Tue Apr 12, 2005 2:35 pm
Location: Albacete - Spain
x 87

Re: The roadmap to 1.9 and 2.0

Post by Xavyiy »

Kojack wrote:The main problems are that the math classes all use _OgreExport so they think they are a DLL, and they all include ogrestableheaders.h which then includes movableobjects and zip files and resources and other stuff that's definitely not needed. Fix those two things and the math code is easy to insert into another project or build as a lib.
+1

Also, the user is able to easily redefine the _OgreExport to an empty label at compiler time so that's not really a big problem.

At the end, I think the top priorities enumerated by relevance are: Next-gen rendersystems, cache-misses/SM refactoring, task-based/data-oriented threading system.
Blader
Gnoblar
Posts: 6
Joined: Mon Nov 22, 2010 9:11 am

Re: The roadmap to 1.9 and 2.0

Post by Blader »

I like everything you have put in the first post CABAListic, however I have a suggestion:

I think it would make more sense to only add features for 1.9 then break compatibility in 2.0 with the cleanup \ redesign compared to breaking compatibility between 1.8 to 1.9.

My favourite part is the cleanup and redesign! Ogre does seem to over-use singletons which is always a sign of bad software design and I am so glad this will be addressed, on top of this it has changed much more from just a "Rendering Engine" as it is trying to manage my resources for me, technically Ogre shouldn't even be worrying about loading files, just getting data in some format and rendering it, not that I'm complaining about that (It can be a modular aspect). On the topic of multi-threading Ogre should not be doing it's multi-threading itself, it should be multi-thread friendly as to allow a person using it to multi-thread with whatever they feel like.
User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
x 218
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Jabberwocky »

Hey Blader -
I appreciate you're here sharing your opinion.
Apologies for disagreeing with most of it. ;)
Blader wrote:I think it would make more sense to only add features for 1.9 then break compatibility in 2.0 with the cleanup \ redesign compared to breaking compatibility between 1.8 to 1.9.
Many major changes are planned for 2.0. But to get there, we need to divide these changes up into smaller tasks. Some of these tasks will be targeted for 1.9. If this means API-breaking changes in 1.9, then this must to be done in the name of progress towards 2.0.

The convension with Ogre has been that API changes are acceptable in new major releases, which 1.9 counts as. It may still work out that the majority of API changes won't occur until 2.0, but I don't think we can proceed with that as a requirement.
Blader wrote:Ogre does seem to over-use singletons which is always a sign of bad software design
Controversial statement. I think singletons are sometimes bad, and sometimes good depending on the characteristics of the code. But lets avoid restarting the singleton holy war in this thread. ;)
Blader wrote:On the topic of multi-threading Ogre should not be doing it's multi-threading itself, it should be multi-thread friendly as to allow a person using it to multi-thread with whatever they feel like.
Disagree. Threading is key for:
1- high graphics performance
2- background loading to avoid graphics hitches and stuttering
3- smooth paging of large worlds
While (2) and (3) could conceivably be considered outside the scope of a pure rendering engine, in practical situations most users who need these features would want Ogre to provide it. You shouldn't need to write your own threading code (a highly expert topic) to achieve good graphics performance from ogre.

I do understand that some people will want more or less functionality out of Ogre. It's understandable some users will want to write their own resource manager or threading code. Wherever possible, this should be addressed by moving things into optional components or compile options, not by yanking it out alltogether.
Image
User avatar
Mako_energy
Greenskin
Posts: 125
Joined: Mon Feb 22, 2010 7:48 pm
x 9

Re: The roadmap to 1.9 and 2.0

Post by Mako_energy »

Jabberwocky,

I think you jumped the gun on Blader's final point. I agree with the retorts of his first two points. However with the third, I don't think he suggested or implied that threading be removed altogether. I personally don't want Ogre to enforce any form of threading on me, even through interface classes, as that implies a forced structure. Putting them into their own components can be acceptable. What I really want is to be able to use whatever exotic threading strategy I want without having to fork Ogre(which I am afraid may happen depending on how these discussions go). As long as that is avoided, I will be happy, and I am sure the same goes for Blader.

I also feel the need to state that I disagree with your word choice of "could conceivably be", where it should be "absolutely are". A "pure" render engine should just render. There are a lot of hobbyists and students out there, and for them providing as many built in tools as possible is an amazing thing. I know it was for me when I started with Ogre. But as I go deeper I find it cumbersome and annoying, as I have to account for Ogre's design in integrating it with other systems.

Just my 2 cents.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: The roadmap to 1.9 and 2.0

Post by CABAListic »

The goal of threading is to parallelise the inner Ogre loop. There is no way whatsoever to do that in an optional component/plugin as this is as core as it gets. But it doesn't mean we have to tie Ogre to its own threading system.

In practice, what I believe would happen is that Ogre's scene update gathers necessary updates and arranges them in some kind of microtasks. This is basically the core part of the threading change and would not be optional. What can be modular, however, is the strategy to execute those microtasks. The trivial strategy just executes them one after the other, not using any multithreading at all. The standard threading strategy for Ogre would manage a pool of threads and distribute the tasks to the pool. And if you want, you can write your own strategy to have Ogre use your own threading system, but you'll have to ensure that your other threading stuff does not block Ogre's microtasks.

Is that agreeable?
User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
x 218
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Jabberwocky »

Thanks for that CABAListic.
That explains in better detail why threading in the core is absolutely necessary. You can't have a modern, high performance renderer without it.

I agree we should keep the threading details abstracted ("threading strategy"), so advanced users can tailor it as they see fit. Hopefully that works for Mako_energy & Blader, although continued discussion is more than welcome.
Image
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: The roadmap to 1.9 and 2.0

Post by spacegaier »

Jabberwocky wrote:Thanks for that CABAListic.
That explains in better detail why threading in the core is absolutely necessary. You can't have a modern, high performance renderer without it.

I agree we should keep the threading details abstracted ("threading strategy"), so advanced users can tailor it as they see fit. Hopefully that works for Mako_energy & Blader, although continued discussion is more than welcome.
+1 to all points!
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
x 218
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Jabberwocky »

Mako_energy wrote:I also feel the need to state that I disagree with your word choice of "could conceivably be", where it should be "absolutely are". A "pure" render engine should just render. There are a lot of hobbyists and students out there, and for them providing as many built in tools as possible is an amazing thing. I know it was for me when I started with Ogre. But as I go deeper I find it cumbersome and annoying, as I have to account for Ogre's design in integrating it with other systems.
Sure, I can agree with that. I guess the real question is whether Ogre should just be a pure render engine, without any of the support systems you generally need to render stuff. Taking the example of resource loading. Agreed, that's not pure rendering functionality. But without it, the audience for Ogre would be vastly diminished to people who are both interested and capable of writing a lot of low level engine code. Not that Ogre is a full game engine, but it does give you everything you need to handle getting graphics on screen.

I think the general consensus is that it's good for ogre to have these support systems, but certain parts should be made relatively easy to replace.
Image
User avatar
zarfius
Gnome
Posts: 367
Joined: Wed Jan 03, 2007 12:44 pm
Location: Brisbane, Australia
x 13
Contact:

Re: The roadmap to 1.9 and 2.0

Post by zarfius »

Let me make sure I understand what we are discussing here. Nobody said anything about getting rid of functionality like resource loading, we just want to separate it from the rendering engine so it can be easily replaced or customised.

Ogre could be a pure rendering engine with plugins for the support systems.
Craftwork Games - hand crafted entertainment.
http://www.craftworkgames.com/
User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
x 218
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Jabberwocky »

Right- resource handling isn't going away, so we can move on from that.

For those of you who want to replace Ogre's resource system with their own, please discuss the changes in Ogre you'd need to facilitate this. Also, much of this code will be community driven, so qualified volunteers are welcome to step forward! :)
Image
User avatar
Mako_energy
Greenskin
Posts: 125
Joined: Mon Feb 22, 2010 7:48 pm
x 9

Re: The roadmap to 1.9 and 2.0

Post by Mako_energy »

CABAListic wrote:The goal of threading is to parallelise the inner Ogre loop. There is no way whatsoever to do that in an optional component/plugin as this is as core as it gets. But it doesn't mean we have to tie Ogre to its own threading system.

In practice, what I believe would happen is that Ogre's scene update gathers necessary updates and arranges them in some kind of microtasks. This is basically the core part of the threading change and would not be optional. What can be modular, however, is the strategy to execute those microtasks. The trivial strategy just executes them one after the other, not using any multithreading at all. The standard threading strategy for Ogre would manage a pool of threads and distribute the tasks to the pool. And if you want, you can write your own strategy to have Ogre use your own threading system, but you'll have to ensure that your other threading stuff does not block Ogre's microtasks.

Is that agreeable?
I'll go into a little bit more detail for my plans, I don't know what others want to do so I can't speak for them much.

I do like the idea of micro-tasks. In fact it's needed for what I want to do as I stated in my post on the first page, which is a Dependency Graph of micro-tasks. To do this we need complete control of the threads, and usage of threads have to be completely lockless/mutex free. I suppose both of those can be controlled within the implementation and can avoid being put into the interface. What I am more wondering is probably something that is a bit too soon to discuss, but what will the "view from 10,000 feet" of the parallelization look like? A long time ago Xavier(the Ogre moderator) posted about how tasks can be parallelized, here. Funny story, it's his rants about multi-threading that caused us to research and choose a dependency graph. In that post he mentions that Transform updates, Animations, Particle Effects, and Shadow Volumes can all be put into their own micro tasks. What else is there to be parallelized? Are all of these tasks more or less independent of each other? Or do a few depend on other tasks? From looking at the existing WorkQueue interface there doesn't seem to be anything to tell it the work needs to get done, but only after other work has been done(i.e. not dependency graph friendly). However I haven't taken a completely careful look at the WorkQueue class just yet. I've only looked at the guts of the default WorkQueue implementation, which I know absolutely won't work for us. So I could have just missed something there.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: The roadmap to 1.9 and 2.0

Post by CABAListic »

Disregard the WorkQueue, I am fairly certain that it isn't suited to microtasking and therefore will probably not survive the threading change.

As for possible candidates for tasks, I can't say right now. I think that's part of the analysis that has yet to be done and is imo too specific for this thread. But the ones you mentioned are probably the prime candidates, and my instincts tell me that they will not (always) be completely independent.
User avatar
Mako_energy
Greenskin
Posts: 125
Joined: Mon Feb 22, 2010 7:48 pm
x 9

Re: The roadmap to 1.9 and 2.0

Post by Mako_energy »

CABAListic wrote:Disregard the WorkQueue, I am fairly certain that it isn't suited to microtasking and therefore will probably not survive the threading change.

As for possible candidates for tasks, I can't say right now. I think that's part of the analysis that has yet to be done and is imo too specific for this thread. But the ones you mentioned are probably the prime candidates, and my instincts tell me that they will not (always) be completely independent.
Whoa. Hadn't realized you were even considering giving an ax to the WorkQueue. I agree with your instincts, and I like the standing plans for the threading re-factor.
User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
x 56
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Klaim »

So basically, the idea would be to have ogre access some kind of thread pool or task scheduler, whatever (or provide it's own if the user want it?), and then Ogre will put in the work to do as tasks objects as needed.

Is my understanding correct?

Because that's what I was assuming.
Blader
Gnoblar
Posts: 6
Joined: Mon Nov 22, 2010 9:11 am

Re: The roadmap to 1.9 and 2.0

Post by Blader »

Jabberwocky wrote:Hey Blader -
I appreciate you're here sharing your opinion.
Apologies for disagreeing with most of it. ;)
No apology needed.
Jabberwocky wrote: Many major changes are planned for 2.0. But to get there, we need to divide these changes up into smaller tasks. Some of these tasks will be targeted for 1.9. If this means API-breaking changes in 1.9, then this must to be done in the name of progress towards 2.0.

The convension with Ogre has been that API changes are acceptable in new major releases, which 1.9 counts as. It may still work out that the majority of API changes won't occur until 2.0, but I don't think we can proceed with that as a requirement.
I don't really get that if it is .Major what is the X. part SuperMajor? Maybe you could clarify that for me. I do think that is potentially a negative for Ogre in some aspects, such as if I were to try and pitch Ogre to my director and included in my report that Ogre may, at any major(minor?) release, break all code associated with it forcing a re-write to catch up to the latest features, they would probably decline it. But that is potentially something for the Ogre team to consider in the future.
Jabberwocky wrote:Controversial statement. I think singletons are sometimes bad, and sometimes good depending on the characteristics of the code. But lets avoid restarting the singleton holy war in this thread. ;)
Well, it isn't really controversial. I agree that there may be uses for Singletons, however nothing that can't be done another, nicer way. The problem with the Singleton design in Ogre is that "All of everything' is accessible everywhere, which helps propagate that same thinking and design.

Plus Singletons and multi-threaded code is not very fun. Yes I know that the Singleton is just a tool in the tool-box that a programmer may use, but there are better ones!

I could probably talk about this for much longer, but I should probably stop now if you would like to discuss this in more detail I would be happy to join you in discussion here.
Jabberwocky wrote: Disagree. Threading is key for:
1- high graphics performance
2- background loading to avoid graphics hitches and stuttering
3- smooth paging of large worlds
While (2) and (3) could conceivably be considered outside the scope of a pure rendering engine, in practical situations most users who need these features would want Ogre to provide it. You shouldn't need to write your own threading code (a highly expert topic) to achieve good graphics performance from ogre.
Yes I agree, but the way CABalistic mentioned is almost exactly what I meant, that is most assuredly agreeable.
Jabberwocky wrote:I do understand that some people will want more or less functionality out of Ogre. It's understandable some users will want to write their own resource manager or threading code. Wherever possible, this should be addressed by moving things into optional components or compile options, not by yanking it out alltogether.
If they are optional components they could potentially be maintained separately to Ogre then right? Although yes you are right, it is most definitely good for the Ogre ecosystem to provide these things already for basic users. One way to do this would well for everyone would be to use branches to maximum effect, letting someone choose between the RenderOnly branch or the OgreFull branch. Something to that effect would be pretty awesome in my opinion.
User avatar
Zonder
Ogre Magi
Posts: 1168
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 73

Re: The roadmap to 1.9 and 2.0

Post by Zonder »

When I think of things been separated in ogre I don't think of them as been in a separate dll's but something that can be safely excluded out of the project maybe cmake options.

Making the math library easy to include in another project would be nice but not separated out.

On the threading side of things maybe make a list of things threads need to be able to do, worry later what library to use I think. Also ogre should still run single threaded as well, it could be that ogre doesn't have an internal threading library it has a ThreadingFactory that can be swapped out but the internal one is single threaded. But I think the team should offer a separate project with a threaded solution that people can optionally use or use as an example to implement there own.
There are 10 types of people in the world: Those who understand binary, and those who don't...
User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 99
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Wolfmanfx »

Threading should also happen on GPU command level so we can multithread everything without taking much care which thread make use of the render system (also DX11 really supports multithreading maybe we should take it into account ).
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: The roadmap to 1.9 and 2.0

Post by CABAListic »

I'm not sure that is a good idea. For one thing, Ogre is not DX11 exclusive, and for another that would require to ensure that Ogre itself is threadsafe. This would not only require to keep thread guards in place in our smart pointers, but possibly also additional safeguards. I'm not convinced this is desirable from a performance point of view?
User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 99
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Wolfmanfx »

here some material

http://beautifulpixels.blogspot.co.at/2 ... mmand.html
http://www.microsoft.com/en-us/download ... px?id=8092

there is much more out there for example http://noesisengine.com/ is also using some kind of "Command Buffer Recording" and he mentioned me that the have performance gain with DX9.
Also our task model should more go into the openCL/CUDA direction so that tasks work on data streams witch special kernels which are optimized for e.g. hardware skinning.
Gamebryo had developed something called floodgate in 2009 i think.

But without a redesign from ground up (Rendersystem) we would just scratching the surface.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: The roadmap to 1.9 and 2.0

Post by CABAListic »

Wait, are you suggesting additional parallelisation potential for the Ogre core, or are you asking that you as the user can access the graphics pipeline (i.e. Ogre) from anywhere? Because for the latter I still stand by what I said.
User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 99
Contact:

Re: The roadmap to 1.9 and 2.0

Post by Wolfmanfx »

Sorry maybe i was not clear enough :) - I mean "additional parallelisation potential for the Ogre core" which should be transparent for the user.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: The roadmap to 1.9 and 2.0

Post by CABAListic »

Ok, we'll keep that in mind. But from a user perspective I still think that we should require all access to Ogre be done from a single thread. That simplifies things significantly, and I believe that in all but the most specific cases it will also give better performance. :)
Post Reply