'Tindalos' (Ogre v2.0) SceneManager redesign

Post by **Praetor** » Thu Mar 29, 2007 11:16 pm

Ok, thanks for the clarification, sinbad. I was thinking of portals as being as generic as connectors, but you gave a pretty specific, limited definition of a portal.

As to what other type of connectors there could be besides portal and some more arbitrary planar boundary, I haven't the foggiest. I suppose the idea here is to put in place a mechanism for extension because based on experience there will be specialized connectors that are called for sometime in the future, regardless of whether we can conceive them now or not. At least, I take it that is your thinking sinbad, unless you actually do have some specific ideas about all sorts of different kinds of connectors.

Chaster · Post by **Chaster** » Fri Mar 30, 2007 12:56 am

That's what I was thinking - that "connectors" would come in different "flavors ' (portal connector, grid boundary connector, etc.) which could be extended if a new design/algorithm comes along, but I'm betting that those 2 types of connectors (portal & grid boundary) will cover 99% of the cases out there...

tau · Post by **tau** » Fri Mar 30, 2007 3:57 pm

sinbad wrote: @tau: I'm not even going to speculate on time scales here. A lot depends on precisely how much time I have to dedicate to it, which in itself depends on finances over the next few months.

I see. I just wanted to know if I need to plan such changes this year Q3-Q4, or the next, 2008 one

Also, such Scenegraph development can save me (and many others) time, maybe even in eq. of one year...
Anyway, I hope (and wish you the best) there will be enough finances for you

jwatte · Post by **jwatte** » Fri Mar 30, 2007 4:44 pm

In all honesty, developing all the art for a game that needs multiple scene graph kinds is going to take a lot of manpower. Most indie games probably don't actually need that kind of power, for that reason alone!

Post by **sinbad** » Fri Mar 30, 2007 4:53 pm

Chaster wrote:That's what I was thinking - that "connectors" would come in different "flavors ' (portal connector, grid boundary connector, etc.) which could be extended if a new design/algorithm comes along, but I'm betting that those 2 types of connectors (portal & grid boundary) will cover 99% of the cases out there...

Yeah, I think you're right, although there's also the 'tube segment' approach that gets used for some racing games too - it's not quite portal and it's not quite a grid connector although it has simlarities with both.

Basically I don't want to limit things to my current assumptions & knowledge since this is a significant redesign that's not going to happen again in the near future, so I'm trying to cover the largest possible base I can and keep my assumptions well under control. I'm well aware I don't know everything, and that new techniques come out all the time so the more flexible I can stay (without hindering the concrete implementations we all know about), the better.

My approach is always to aim for the most flexible design with the fewest assumptions, and to crawl back from that only when I have to given any practical issues. I think that can lead to better results than starting with a fixed idea of the implementation and presuming I understand all the current and future options that might be used.

Dom · Post by **Dom** » Fri Mar 30, 2007 7:31 pm

It's not really that related to the topic, but sometimes it seems precision pops up as subject. You probably already know this article, but in case not: http://home.comcast.net/~tom_forsyth/bl ... sion%5D%5D

For little chunks this might not matter but maybe for big ones or meta-chunks.

Post by **Praetor** » Fri Mar 30, 2007 7:46 pm

That's one of the issues sinbad wants to address directly with this. Each locality can have its own coordinate system to you could break up your "universe" into chunks that retain reasonable precision.

Chaster · Post by **Chaster** » Fri Mar 30, 2007 9:28 pm

Since we're deviating a bit off the well discussed connector topic...

Sinbad, I propose that the coordinate system be modified so that it is "relative" to an arbitrary point. This will go far in assisting with the problem of round-off error and I suspect that you already know this since you talk about multiple local coordinate systems tied together with macro coordinates.

Also, I would like to see the camera class allow for an arbitrary number of extra culling planes. I'm not suggesting this just because it would help my portal scene manager implementation (it would) but because I think it would be useful in other respects (since hardware clipping planes are quite common, but not necessarily universal).

Also, what do you think about the idea of "requiring" cameras & lights be attached to scene nodes like other entities? I think it would actually streamline some of Ogre's internals and make for a more consistent user interface, but I know others may disagree..

Chaster

jwatte · Post by **jwatte** » Fri Mar 30, 2007 10:14 pm

What we do at work, where we have an earth-size planet, is to store entity locations (including camera) in double precision. Meshes, however, are stored in single precision. Everything is then rendered in camera-relative coordinates. Because the camera-relative coordinate is in the range [-15000..15000] or so, it turns out to be free of precision and jitter issues. We also simulate in double precision, which means we don't have to handle the case of a simulated entity rolling from one cell to another in any special way.

The main draw-back, from an Ogre point of view, is that our space is euclidean -- we can't do overlapping spaces, which you can do with certain portal solutions. However, if you have overlapping spaces, you have to be careful to never go from raw coordinate to find a space, as you may find more than one space.

Post by **Praetor** » Sat Mar 31, 2007 12:18 am

That blog was interesting, Dom. I've often considered fixed precision, but since I've never run into scale problems it's never been a serious consideration. Interesting read, though in some ways the guy seemed a little too intense.

Post by **sinbad** » Sat Mar 31, 2007 3:00 pm

Yes, the design has always been to have all positions within localities to be expressed relative to a local origin. Thus, operations and comparisons within that locality are subject to fewer precision issues, including the fact that the view matrix will be relative tot he locality and thus precision is regularly reset according to the locality size. Comparisons between localities will obviously have a higher imprecision but since you're only likely to want to compare neighbouring ones the extent of this is clearly well-boxed. The intention is a much-generalised version of the technique presented by Peter Freese of NCsoft in GPG4.

Arbitrary culling planes is an interesting one - we actually already support this at the RenderSystem layer but I've avoided promoting it because the vast majority of cards only support a max of 6 hardware accelerated clip planes. For just culling (not clipping) it would be ok though I guess.

Yes, I already had the removal of cameras & lights being freestanding on my personal list

The reason is that clear ownership becomes totally mandatory and it needs unifying. The other reason is because I intend making node movement / resize events the primary way a SM knows how to adjust its spatial system rather than having a tightly owned custom SceneNode. So unification is necessary.

I have a ton of scrappy notes and even more things in my head but a few unexpected things came up last week so it never got organised. rest assured I will be writing it all down properly and organising it into a design document that everyone can review - I don't expect to start coding on this for a little while yet, the design has to be a lot more solid before that happens.

satol · Post by **satol** » Sat Mar 31, 2007 3:43 pm

The locality system does not help precision much, when the change of scale is required (from miters to parsecs), I believe.

Post by **sinbad** » Sat Mar 31, 2007 4:05 pm

Explain?

The locality system helps as much as any system can with extremes in scale in a large, mostly flat structured world. There will still be a double-precision option but because this is useless at managing GPU resources, the approach should be that if you're finding the localities aren't giving you enough precision, you need to consider smaller and more numerous localities.

The locality idea is that you'll only see / need detail when it's close to you anyway - thus if you lose a small amount of precision far away you won't notice because it's too far to see, or objects are far enough away that detailed collision etc is irrelevant.

You've suggested using a local view matrix before (ie local to a SceneNode tree), and the locality concept is largely the same - it's just more explicit about what 'local' means in the more usual case of a relatively flat world tree, and for all comparisons (not just camera calculations). Being local in the sense of a subtree of the world is pretty difficult to apply to a general case world.

satol · Post by **satol** » Sat Mar 31, 2007 11:54 pm

The locality system, if I understood the concept correctly, separates space into the grid of constant cell size. A grid cell of several kilometers in size will be too small for the solar system scales. The way I suggested allows to exploit the nested local coordinate systems, so it does not have such problem. One need to take care of moving the camera between nodes though.
You need to extend the grid locality somehow to allow nested grids, then it would work always, I am not sure if it worth doing, as far as there is a simple approach.

Post by **sinbad** » Sun Apr 01, 2007 4:09 pm

I see what you mean. Well, I never in fact said that localities had to be in a grid, and I actually did say that they can overlap (and implicitly therefore can be nested). The reason for that was actually indoor localities within a larger outdoor one, but your space example is a good one.

So in the case you describe you could have one locality within another - all that's missing is a relative scale between them. Good idea, I will add that to my notes.

I'm glad because this actually vindicates my insistence of not assuming the only connector types will be portals and grid boundaries.

satol · Post by **satol** » Sun Apr 01, 2007 4:32 pm

Nested grids should be able to move relative to each other as origins can move (planets move, an the grids on the planets should also).
The portal SM (a rocket) can move inside other SM (space).

Post by **sinbad** » Sun Apr 01, 2007 4:47 pm

You don't want to make this easy for me, do you?

Ok, that's in my notes now too.

Post by **Praetor** » Sun Apr 01, 2007 5:26 pm

Basically, sinbad, please make the most amazing scene management that has ever existed. It should support perfect extensibility and solve any precision and nested scene problems ever. Thanks.

This is turning out to be a taller order than I ever imagined.

Game_Ender · Post by **Game_Ender** » Sun Apr 01, 2007 5:43 pm

Oh, and I think we want it for free and yesterday

. Good luck on getting the design worked out. I myself should do more paper work before my fingers hit the keyboard, would probably end up saving me time.

satol · Post by **satol** » Sun Apr 01, 2007 6:50 pm

I vote for more abstraction instead of complexity.
I do not want the Ogre to provide a perfect Scene Manager - there is no one exist and I do not need one. I want more abstraction in there, so I could override the behavior to have classic transformation inheritance, which would solve for me precision issues perfectly. I may be do not want to use the Ogre Scene Managers at all, only the material system and rendering driver abstraction. It is not possible to separate however in Eihort, because there is dependence on the scene managers everythere in the Ogre.

raicuandi · Post by **raicuandi** » Sun Apr 01, 2007 8:46 pm

[ot] Doesn't it, like, 'hurt your head' when you try to figure out all of these new design decisions, that probably nobody has done before, and trying to do them all: satisfy everybody, make sure its abstract enough to be extensible in the future etc.? I'm serious...

/edit: suggesting abstraction over over-complexity too. I use it in my GUI system, and I introduced features that would otherwise take mounds of code/complexity in classic "static" systems like CEGUI.

Post by **Kencho** » Mon Apr 02, 2007 3:28 am

raicuandi wrote:[ot] Doesn't it, like, 'hurt your head' when you try to figure out all of these new design decisions, that probably nobody has done before, and trying to do them all: satisfy everybody, make sure its abstract enough to be extensible in the future etc.? I'm serious...

Not really. There are a lot of design patterns and well-proved solutions out there. Experience does the rest (experience that, by the way, I'm sure Sinbad already has

).
About satisfying everybody, with the ideas exposed here, it will be great enough to cover the 99.9% of the users spectrum

Post by **sinbad** » Mon Apr 02, 2007 3:47 pm

I've renamed this thread since I've decided that these changes will go into Ogre v2.0 aka 'Tindalos' and not v1.6 aka 'Shoggoth'. I'll be developing it in parallel but the interface changes are so significant as to warrant insulating people from them if they just want the other 1.6 enhancements - the Google Summer of Code projects will be part of 1.6 for example, among other things. I also can't think of any other change that deserves a v2.0 moniker more than this one.

As for abstraction vs complexity - yes in many ways abstraction can be used to reduce complexity if done right. The existing scene manager design originates from the very first version of Ogre and has suffered growing pains since then, it's time to not so much wipe the slate clean, but to rethink everything from first principles, given what I've learned since that time (which is a lot, thanks in many ways to the diversity of this community).

Here's something else to lay on you: in Tindalos I'm almost certain to be removing the need to use SceneNode at all. There has to be something which provides transform information to the renderable objects, but it won't have to be a SceneNode, although clearly that's a rather useful structure to use for many people. By decoupling this such that objects only need a generalised 'transform source', you get quite a lot of benefits. One of which is being able to tie Ogre objects directly into your own external world management system. Having an externally managed world with just incoming sync events to drive things like SM spatial structure updates was something I wish I'd done at the start, so now I think it's time to do it. It's made possible by the fact that custom SceneNodes will no longer be the mechanism for updating custom SM structures.

The other thing will be a redesigned event system. I plan on funneling all callbacks via a generic event system which can be listened in on and extended by custom plugins so new events can be added without having to bind to specific custom interfaces. External world management systems will be required to spit events into this system to notify of object transform changes (and yes, this means separate threads can be used for each purpose).

The scope of this review is going way beyond just SceneManager techniques, I'm really looking at the whole world management & integration systems very closely looking for ways it could be done better. Clearly it's very daunting a task and will take a while, so be patient with me please.

You should know by now that I prefer to take a bit of time over these things and try to cover all the angles.

Post by **Praetor** » Mon Apr 02, 2007 3:58 pm

Interesting. I look forward to seeing such a generic event system. I have a notion of generic callables in odyssey, but it is so coupled with the rest of library it would be hard to get out.

Code: Select all

Callback call1, call2;
call1 += callable1;
call2 += callable2;

call1(0.5f, this);
call2(1, 2, 3, 4);

Notice the lack of type information...

Are you saying that one method for managing a scene in 2.0 will be event-based? As in, the scene just sits there dumbly, and you control all its aspects by externally raising events (like, entity moved or rotated, material switched, etc.)?

raicuandi · Post by **raicuandi** » Mon Apr 02, 2007 4:01 pm

Here's something else to lay on you: in Tindalos I'm almost certain to be removing the need to use SceneNode at all. There has to be something which provides transform information to the renderable objects, but it won't have to be a SceneNode, although clearly that's a rather useful structure to use for many people. By decoupling this such that objects only need a generalised 'transform source', you get quite a lot of benefits. One of which is being able to tie Ogre objects directly into your own external world management system. Having an externally managed world with just incoming sync events to drive things like SM spatial structure updates was something I wish I'd done at the start, so now I think it's time to do it. It's made possible by the fact that custom SceneNodes will no longer be the mechanism for updating custom SM structures.

So its like I'm asking another "master"-class "at what position/what orientation etc. do I render this entity?"