Paging as a core feature - design notes

[sinbad]

Hi folks,

Thanks to a co-operative project that I'm doing with a client, I'm going to have the opportunity to spend time doing 2 rather important things over the next couple of months:
1. Implementing a brand-new terrain system
2. Promoting the concept of paging to a core feature

This post is about #2 - I'm not going to talk about the specifics of the terrain here, but I mentioned it simply because I wanted to put it in context. It's going to be a paging-aware terrain, but I absolutely did not want to make the paging specific to terrain only. This deserves to be a core feature, so step 1 is defining what that paging system will look like.

Firstly, a few principles. As you know I like to keep things flexible, not make any assumptions I don't have to, and promote extensibility. That's why you'll probably look at the designs I'll post here and think they're overkill - but the intention is to provide a good base for the future. The kind of paging I want to see will:

1. Not assume that the system ever has complete knowledge of the entire world - new parts of the world must not just be capable of loading in based on a predefined structure, but of being 'discovered' dynamically too
2. Allows multiple strategies for decided the lifecycle of pages - as-the-crow-flies distance is an obvious one, but we must also allow for other approaches (particularly for dense, occluded scenes it might be region connectivity and occlusion stats)
3. Treats the paging strategy and content of the page as orthogonal concepts, ie you can mix and match them
4. Does not assume regular sized pages, or that pages occupy unique, non-overlapping regions in space
5. Supports varied structural page setups - grids for terrain perhaps, but 'snake' like areas for cavern systems.
6. Transitions must be possible between pages without type-specific knowledge (vital for supporting varied scenes, e.g transition from cave system, to 2D terrain, to space)
7. Pages must be able to be loaded in a separate thread, with the same modes as available for resources (full threading or semi-threading).
8. Pages should be able to have multiple LODs in which the physical content changes. Content within the page may have LOD of its own, but we should also allow the physical content to change too at a higher level
9. For now, no support for page-relative co-ordinate systems. To cope with precision in large worlds, double-precision coordinates and camera-relative rendering can be used. Page-relative co-ordinate systems require a larger scope change which will be reserved for a later version.
10. In general, paging support must not require major changes to SceneManager, those are still reserved for 2.0

So, this is pulling in a number of concepts from things that were scheduled for Ogre 2.0, but ring-fencing them so that they can be incorporated without requiring the wholesale changes to the SceneManager that will be required for that (hence no page-relative coordinates).

Here's the first design diagram I've come up with:

paging_design1.gif (18 KiB) Viewed 21460 times

PageManager
This primarily a central registration point for extension classes, such as the PageStrategy, PageContentFactory, and the place to access the currently loaded content (PagedWorld). I haven't decided yet whether I will hang this off Root in the core, or make it a separate library which a user references if they want.

PagedWorld
This is basically the data-driven part of the structure. A PagedWorld is a collection of world content which can be loaded from a file or defined in code. What's important is that it does not need to create everything that's underneath it at load time - minimal set up is creating the PagedWorldSections (see below), and optionally PageEntryPoints if there are 'spawn points' that need to be defined (this is important if it's not possible to derive what page needs loading simply from a camera world position). With these in place, requests can then dynamically load the rest of the parts that are needed on demand.

PagedWorldSection
This is a section of the world which conforms to a particular paging strategy. For example, an outdoor 2D terrain section (and by 'terrain' I means structurally 2D, not that it defines terrain content) might use one strategy, bounded in one way, a space section of the world might use another strategy, a complex indoor area might use another one. Only a high-level description is held here - a section ID within the world, general bounds information, an optional list of predefined entry points (with just a position and a page identifier - especially important if the strategy cannot translate a position directly into a page with no context).

PageStrategy
This class is responsible for determining how pages are loaded and purged, and optionally deriving a page just from a world position (and a PagedWorldSection). Whenever a camera moves, the PageStrategy is responsible for deciding how that affects page requests, within the context of a PagedWorldSection. This might be a grid system, it might be a case of using connections between pages to determine traversal depth. Whatever the case, given either a global position, an existing page, or a PageEntry point, this strategy must be able to determine what pages need to be requested or disposed of.

PageStrategy is designed to be user-extendable.

Page
This class collects together all representations of a page of data. It defines the page, but doesn't actually contain any data of its own. Rather, it just contains a list of LOD levels and a reference to a LOD strategy. A page has a unique ID within the world section.

PageLOD
This is to allow the content of a page to load more than once if need be. You may, for example, want to have a page which loads in multiple stages, as a very simply distant definition with basic content, and as more detailed definitions closer up. I intend to re-use the pluggable LodStrategy classes to represent the transition rules.

This now defines the uniqueness of our units of loading for pages: "WorldID:WorldSectionID:PageID:PageLODIndex". I will probably put a pluggable translator system in place so that takes these numbers and translates them into a unique resource to load from - which could be as simple as a file ("myworld_sec1_p1234_0.dat"), or could be translated into an offset in an existing packed resource file or something.

PageContent
While PageLOD is the unit of loading, you may still want to include multiple types of content. For example in a page you might have a chunk of terrain, some static geometry, some custom user data (like trigger regions, world objects etc). Therefore within a PageLOD you can have multiple PageContent instances, which all get loaded in together but each of which can have custom data formats and routines for bringing them online. At load time they will be instantiateed through a factory system based on named loaders.

Each of these PageContent types is free to do their own version of LOD too - for example if the content includes creating some Entity instances and they have Meshes which use LOD, then that will still apply. For the terrain, each page will have it's own subdivisions and LOD. But the assumption is that this is all in-memory LOD, suitable for lightweight dynamic adjustment, whilst the higher-level PageLOD is much more heavyweight and actually changes the content, changing the memory footprint etc (and also being potentially threaded). That's why there are 2 types of LOD here, it's not duplication.

PageContent and PageContentFactory are designed to be user-extendable.

Phew....
Ok, that's one of the largest posts I've ever made. I hope that looks good to people - it's still very much subject to refinement so nothing here is set in stone, but I've spent quite a bit of time thinking about the overall structure, what kind of functionality I want to support, and where the extension points are so I'm pretty happy with so far. Comments are welcome!

[sinbad]

Things I forgot to mention:

1. Yes, this means I'll be defining a new 'world' file format, although actually it's a meta-format, since the data required to build the PageContent instance is designed to be variable. I will, however, be definining content implementations for terrain and probably a bunch of regular scene objects, so this could be seen as replacing dotscene to a degree, although there's no reason why dotscene couldn't still be used too. By nature the format I'm thinking of is a binary one, with possibly a new XML serializer (although the pluggable content data will limit how much that can be broken down beyond CDATA sections)
2. Paging will be entirely optional. PageContent will at the end of the day be inserting data into a SceneManager. People who don't need paging will not be forced to use any of this, which is why I'm considering making it a separate (but still core) library.
3. PageConnection is about knnowing about the neighbours of a given page - this is particularly important when going between PagedWorldSection instances. I'm still a bit wooly about the specifics, but it's a lot like the zone connections in PCZ, just much larger granularity
4. I probably need to think about PageLOD fading / smooth transitions. Not top priority for the moment, it's just important to have the concept of multiple PageLOD levels in there for now

[Noman]

Interesting stuff!

Is the terrain system going to be open source as well or is that private?
If such a system is written, and no example paging geometry plugin is available on top of it, it will be hard for users to create their own paged systems, kinda like giving the SceneManager interface without specific implementations...

Having a bit of a hard time deciding whether I think paging should go into the core or as a plugin... Close call...

Also, can you describe how you vision the interface?
(Which subclasses will the terrain system implement etc)

[lf3thn4d]

WOW! This is really cool!

1. Would pages be able to load up and cached in memory before it is actually used?
2. When does a page gets unloaded?
3. Is it possible to specify strategies that works like portal?

This actually looks very good. I can see many uses for this. Including the paging geom add-on. I believe PCZSM can also benefit from this.

[KungFooMasta]

Very exciting! Also consider the case where there is paged terrain and physics involved. For example would the scenario where a large boulder on top of a hill (far from viewing distance) be able to roll down terrain into viewable regions? Maybe its not possible to do paging with this kind of situation. Just thought I'd throw this out there as an idea to think about. I think this should be a core feature. Would a paging solution worsen performance if there is no need to do paging? (ie a paging system has a world that is small enough that it fits on a page)

[Praetor]

Would a paging solution worsen performance if there is no need to do paging? (ie a paging system has a world that is small enough that it fits on a page)

If you know that the world is one page large, then don't even initialize the paging system. Instead, continue to directly use the scene manager as you are now.

@sinbad This is really critical I think for the future. We were looking at a lot of this for 2.0, but it would have to be started some time and I think it's great that it is starting now. Having this functionality in the core puts Ogre up one more level on the power scale. It's definitely true that as we move further into general scene management we start to approach that blurry line of "are we trying to do too much?" But I wouldn't worry here, I think we're still well on the right side of that line with this.

You didn't go over the PageRequestQueue but I'm guessing that's the point where applications "hook" into the system to intercept page load requests. If I were implementing this system in an application, what's the minimum I would need to get a paging system running? Meaning, what classes do I have to subclass, or listeners attach? I'm guessing the main components to interact with for control are the PageWorld and the PageContent. Am I getting the general idea?

[Jabberwocky]

Awesome and exciting news!
Thanks to both you and the client for arranging this to be included in Ogre3D, rather than proprietary work.

---------------------------

I have some questions about the world file format.

Ok, so the file data for a page can include both:
1- graphical data (can be interpreted directly by Ogre + SceneManager /Paging plugins)
2- game objects/data (user defined, extensible. Not interpreted by Ogre.)

From a client perspective, there's some ambiguity here, because you may not want to use the same policies to load the graphical data as the game objects.

Loading Graphical Data
The PageStrategy in most cases would be based on visibility. Generally, we don't need to load graphical data until we need to draw it. And we can unload it when it's no longer visible. An example is terrain heightmap data, which we're probably free to page in and out of memory as it is needed for graphical purposes.

Loading Game Objects
Game objects may need to be loaded before the corresponding graphical data within the same geographical region (I'm specifically avoiding the word "page" here). For example, perhaps I need to be able to communicate with a non-visible NPC. Or perhaps I have a UI which displays statistics on all the game objects in the world, regardless of their visibility. In either case, the game objects must be loaded and unloaded independently from the graphical objects. A different way of looking at it is that we may need to keep around some kind of state for our pages beyond what must be loaded and unloaded from a static file for graphical purposes.

Saving PageContent
Is there any notion of saving PageContent when it is unloaded? For example, I may want to maintain state about how a page has changed. For example, the player blows up a building in a loaded Page. I want to maintain this state for the next time the page is loaded.

Some possible solutions to these problems:

Solution 1: Not in Ogre's scope.
It's up to the client to load and save any game data which doesn't conform to the page strategy of the graphical data. This game data would come from a separate file that Ogre never touches or loads at all. Ogre's world file format is purely for non-persistent data which will be loaded and unloaded, usually based on a visibility strategy. Ogre's only responsibility is to provide the appropriate hooks / notifications so the application can do any app-specific work when a page is loaded and unloaded.

Solution 2: Separate Page Strategies for game vs graphical data.
Split the game data and the graphical data into two separate pages. For example:
- file page1_geom.xml contains all the heightmap and static geometry data that will be loaded and unloaded based on a visibility policy.
- file page1_game.xml contains all the dynamic game objects which may exist in this region, but can implement a separate PageStrategy.

Solution 3: Support different load states for a page, maybe tied into LOD somehow
- game data within a page can be loaded and unloaded separately to the graphical data, perhaps being totally persistent.
- the graphical data, or graphical representation of game objects are loaded and unloaded based on visibility

Solution 4: Divorce the idea of page data from a file
- allow the application to define it's own page class, which is responsible for responding to requests from the PageManager.
- the page class can respond to load and unload in whatever way it likes
- the page class can retrieve data from files, a database, or generate it algorithmically.
- the page instance is not necessarily destroyed when it is unloaded - it's free to keep any persistent data around which it requires
- the page class must present paging data to the page manager in a well defined format, perhaps equivalent to the binary data format of a world data file.

There's some overlap in those ideas, but I'm just kinda thinking out loud here.

Things get a tricky because of how interwoven the graphical and game data can be. For example, a tree could easily be considered purely graphical data (position, orientation, mesh). But a tree with hitpoints, a growth stage (starts as a sapling and grows over time), an occasional rustling sound, physics, and an area-of-effect magical healing power is certainly both graphical and game data. It would be nice to keep all this data bundled together in the same file/class/database. But that complicates the loading and unloading strategies as I've outlined above.

I guess my question boils down to this. Are pages necessarily made out of (and limited to) file-based static data (such as a terrain heightfield and tree distribution map), or do they support some kind of dynamic game state (such as movable object positions, and game-system related data)?

Maybe it would be useful to walk through an example of an existing game with a large world, like Farcry2 or Fallout3. As the player travels across the world, and stuff appears and disappears on the screen, which parts are handled by the PageManager, and which parts are not?

[sinbad]

Is the terrain system going to be open source as well or is that private? If such a system is written, and no example paging geometry plugin is available on top of it, it will be hard for users to create their own paged systems, kinda like giving the SceneManager interface without specific implementations...

Everything will be open source, the same core Ogre license. The initial priority for concrete implementation will be the terrain-related classes, since that's my client's primary concern initially, but in order to maximise the investment in Ogre here I'm trying to make this a much more general, strategic chunk of functionality. I've reached a co-operative agreement with my client so that everyone can benefit long-term. It's great when you can find customers that appreciate the benefits of operating this way!

Having a bit of a hard time deciding whether I think paging should go into the core or as a plugin... Close call...

Well, I'm thinking a separate (but still in the Ogre core) utility library that users link to in addition to OgreMain, and tie the two together. So it's not really a plugin per se, since users will be using it directly.

(Which subclasses will the terrain system implement etc)

Take a look at the diagram for classes with 'Terrain' in them Essentially the terrain will provide a connection to the paging system via TerrainPageContent primarily. Also GridPageStrategy is very closely related, although it can be used for more than just terrain so will not be directly associated.

1. Would pages be able to load up and cached in memory before it is actually used?

Yes, the same kind of system as Resources is envisaged, with an ability to either fully load (fully threaded, including rendersystem) or simply page into memory (semi-threaded, via prepare() like functionality) in the background.

2. When does a page gets unloaded?

When the PageStrategy says it should. This is entirely customisable - it might be a simple grid distance, or it might be when the links from the page any camera is in is more than 'N' connections (plus any other metrics you might want to use) away.

3. Is it possible to specify strategies that works like portal?

That's precisely what PageStrategy is for. I put in a placeholder called 'ConnectivityPageStrategy' to show the ability to extend into this kind of area. To begin with I'll be starting with the grid-based functionality, but I wanted to make sure there was room to extend to other types.

Very exciting! Also consider the case where there is paged terrain and physics involved. For example would the scenario where a large boulder on top of a hill (far from viewing distance) be able to roll down terrain into viewable regions? Maybe its not possible to do paging with this kind of situation. Just thought I'd throw this out there as an idea to think about.

This is why there is PageContent (which can include any content you need to load, might not even be graphical), and also PageLOD. At some point you're going to have to say that you'll freeze the physics at a certain distance, you can't simulate the entire world all the time. You could do that via a PageLOD. I'm actually thinking of moving the PageLOD below a PageContentType class or something like that so that multiple LOD strategies can be specified for different content. But anyway, all paging systems have to at some point deal with what happens with dynamic objects, and the PageLOD is designed to give you hooks to make that decision. At some point you will have to accept that you will lose track of a physics object once it gets a certain distance away - remember, we're talking pretty large distances here.

Would a paging solution worsen performance if there is no need to do paging? (ie a paging system has a world that is small enough that it fits on a page)

Well, you just wouldn't use it then. Like I say, the paging is supposed to be something you use if you want, it's not a required interface to use (hence why I'm increasingly thinking of making it a separate lib from OgreMain).

@sinbad This is really critical I think for the future. We were looking at a lot of this for 2.0, but it would have to be started some time and I think it's great that it is starting now. Having this functionality in the core puts Ogre up one more level on the power scale. It's definitely true that as we move further into general scene management we start to approach that blurry line of "are we trying to do too much?" But I wouldn't worry here, I think we're still well on the right side of that line with this.

Yes - this doesn't go to the extent of 2.0; there's no page-relative coordinate systems, I'm not attempting to tackle the fully threaded scene management yet, and essentially the entire paging system is just a 'feed' to the existing scene content structures. But the intention is to establish a base feature set which works with the way the SceneManagers currently operate but yet still delivers a lot of value. When the project came up and I really sat down to start thinking about it, I increasingly felt I could deliver a decent 'stepping stone' without having to undermine the 2.0 vision, but also without having to change the main interfaces too significantly. Being able to pull off a more manageable chunk like this and digest it is nice, and the fact that it's partially funded also means I can dedicate more time to it than I would be able to otherwise.

You didn't go over the PageRequestQueue but I'm guessing that's the point where applications "hook" into the system to intercept page load requests. If I were implementing this system in an application, what's the minimum I would need to get a paging system running? Meaning, what classes do I have to subclass, or listeners attach? I'm guessing the main components to interact with for control are the PageWorld and the PageContent. Am I getting the general idea?

The intention is that the PageStrategy pushes things onto the request queue. The idea is that this is mostly data-driven, since by nature all of the data is going to need to be 'discovered' on demand from previously saved data. Real-time construction will also be possible, but essentially in that situation you're overriding the PageStrategy because you're deciding when a new page gets constructed, and so this isn't the 'normal' usage. Assuming the data is already defined, then you would simply load a base world file, which would define the general world sections (which are defined by their type of paging - you could just have one here), which would establish a PageStrategy that should be used. You then have to place a camera, and the PageStrategy will decide, in conjunction with the world section which must include some strategy-specific instance data, what page(s) to request. So if your data is already created, you really only *need* to talk to the PageWorld and maybe the PageEntryPoint if you want a camera position suggestion. In practice however, you will probably have a need to drill down into the PageContent too if you want to do specific things like get a list of objects that are on the page, or perform terrain enquiries, or modify data in the case of authoring. All PageContent sublcasses will be required to be able to serialize themselves so that they can be saved / loaded as part of the binary page data.

@Jabberwocky:
Ok, firstly for global data like you describe, I wouldn't be using a paging system at all. Or, if you do still need some kind of paging, then you can use a different PagingStrategy to reflect how you want that data to be maintained. I made a specific design assertion that multiple PagedWorldSection instances can exist at one time, and that they do not need to be spatially separate. In addition, PageContent can be any type of content at all that needs to be paged in and out of memory, not necessarily graphical content. Finally, as I mentioned above I think I'm going to make PageLOD specific to content type so that per page, once you've decided that you're interested in the page as a whole, you can decide that each set of PageContent within it changes LOD based on different criteria. Essentially the PageStrategy is in charge of deciding when a Page is of interest at all, and therefore loads some base page definition which will include content types and LOD strategies for them. The LOD within the page (and now content type) is then required to decide when LODs of that content change, and what the effect of that will be.

I didn't make it clear but requirements for saving will be a part of the API at all levels.

When I talk about 'files', in regard to the world, the pages and the page content, I'm really talking about a load hook. I'm not sure where to put the implementation for this yet, but I the request will be placed on the PageRequestQueue and 'something' will process it. At the base level options for synchronous per-frame checks and a thread to do this are obviously on the table. When a request is being processed though, there must be a flexible way of mapping the request to an implementation - whether that's looking for a separate file for each request, or pulling data from a stream, or defining it procedurally. These implementations should be able to load / build the data then queue the result back for attachment to the world (which will be synchronous, and if we're using semi-threading then no GPU resources will be created in the separate thread).

Regarding dynamic state, I think it's important to think of the page system as a manager of 'snapshots' of data. Once a page is loaded, it may well create dynamic objects, which will just be regular MovableObjects as far as Ogre is concerned. KungFooMasta's point about a boulder rolling across a page boundary is illustrative - unless the page in which the boulder starts was already loaded, the boulder could not roll into an active page because it doesn't exist yet. This is an unavoidable simplification and I think pretty easy to understand, and people will structure their paging to make this as least noticeable as possible - what's more difficult is unloading. If your page spawned a dynamic object and the page get's unloaded, what should happen to the dynamic object? Well, essentially this depends on how mobile the object is. If it moves a bit, but generally stays in the local region, then destroying it when the page is removed probably isn't a problem. If you can carry the object all the way across the world with you (e.g. you drive a vehicle that was spawned in at a page), that could be more of a problem!

I had earmarked ObjectPageContent as a placeholder for spawning general MovableObject instances in a page. My suggestion would be for the default behaviour to be to destroy the objects when their page is unloaded, but to have the ability to detach an object (or perhaps its parent node) from a page, at which point it is essentially immortal until you manually destroy it, just like a regular Movable. I could also allow an object to be re-attached to a page (e.g. when you put it down and walked away) so that when that page (might be a different one) gets unloaded, the object then gets destroyed. Of course, this would require the user to be aware of pages, but that's how I envisage it being possible.

Essentially each PageContent subclass is responsible for how it manages the objects it creates in response to paging activities. Those objects do not necessarily have to remain attached to the paging system forever.

Hope that covers everything!

[Assaf Raman]

There is so much text on this thread and I didn't have the time yet to read it all. Forgive me if I am writing something that was discussed.
From what I can see it is worth your time to learn this d3d sample: "ContentStreaming Sample".
I think it sums up much of the modern solution for good performance with streamed terrains.

[jacmoe]

I am just loving it!

This is a very important move for Ogre3D, because it would mean that the various terrain tools will be able to offer Ogre exporters.
And that would mean that Ogre becomes an even more viable choice for professional game developers relying on paging terrain support.

I know the Grome team has been wanting to make an Ogre exporter for some time now, but didn't find any official Ogre paged terrain plugin, so they put it on hold, and made one for Torque..

So, this is going to be bloody nice.

[tuan kuranes]

Brilliant design !

A few remarks & questions :

1) would it be possible to colorize/emphasize user "extendable"/"mandatory extendable when paging" class in the graph ?

2) if user provide page content, will he be able to tag mesh/renderables as world fragment ?

3) just to be sure, we can have mutiple pagelod at the "same time" for the same page ?
Does "Camera" still belongs to scenemanager ?
Will renderables be shared/shareable across a page world, or renderable will be per scenemanager/per pagecontent ?

4) would it be possible to have multiples pagemanagers?

5) for sure, all those "user extendable" introduces many caveats (a "lib", Root core)
Why not the just only the "Event" way ? I guess there will be PageContentEvents, PageLodEvents, etc. anyway ?
With giving some way so that user can generates event (once he has his data ready )

[sinbad]

1) would it be possible to colorize/emphasize user "extendable"/"mandatory extendable when paging" class in the graph ?

I can certainly colour code the extensible bits, I'm not sure what you mean by 'mandatory extendable' though - the idea is that if you use the content types and strategies already provided for (which for terrain, will already be there), then you don't have to extend anything if you don't want to.

if user provide page content, will he be able to tag mesh/renderables as world fragment ?

Essentially the PageContent implementation can do absolutely anything it likes, so long as it can implement load/unload/save. The intention is not to assume anything about what the 'content' might be.

just to be sure, we can have mutiple pagelod at the "same time" for the same page ?

That was what I was alluding to about 'page lod transition'. I haven't quite decided how to handle that yet (the base terrain won't need it, since only internal chunk LOD will be needed initially) but yes I would like to have the concept of an overlap between active LODs to allow for some kind of smooth transition.

Does "Camera" still belongs to scenemanager ?

Yes, it's important that nothing in the way SceneManager functions is changed here.

Will renderables be shared/shareable across a page world, or renderable will be per scenemanager/per pagecontent ?

Again, nothing will change from the current setup here. So standard MovableObject instances will be specific to SceneManagers, but users are free to create custom Movable/Renderable subtypes that are owned however they like, just like now.

would it be possible to have multiples pagemanagers?

In theory yes (since I intend that the user will instantiate PageManager themselves rather than it being a singleton, reinforcing it's 'utility' pattern), but in practice I'm not sure why you would need to. The PageManager is literally just a registry of strategy implementations, content factories, and loaded worlds.

5) for sure, all those "user extendable" introduces many caveats (a "lib", Root core)
Why not the just only the "Event" way ? I guess there will be PageContentEvents, PageLodEvents, etc. anyway ?

Mainly because these events cannot be predicted - because if you want to implement another way of deciding when a page is created / destroyed then the event that would trigger this is based on entirely different criteria, and therefore we cannot predict when the event should be raised. There will certainly still be events generated, but for full flexibility we need to provide pluggable ways of raising those events in the first place, hence the patterns.

Also, I was looking for a declarative approach rather than a procedural one. I think this fits with the most common requirements for paging. However, if a user wants to inject pages into the system themselves based on low-level information like camera movements instead of implementing a PageStrategy / PageContentFactory, then that should be possible too, but really they could be doing that already

[Jabberwocky]

Sinbad - thanks for your clear explanations and detailed responses.

Sorry for another monster post. No response required - I think I'm largely regurgitating the info you've provided.

----------------------

I want to run through a usage case, and how this would translate into different PageContent and PageStrategy types.

Usage case:

I'm making a fantasy RPG game, which entirely exists on a single sprawling 2D terrain. I have a particular region of the world I want to make into a Page. This page contains:

1. terrain

• e.g. heightmap file based
• loading of heightmap data for this page is implemented via a single TerrainPageContent instance.
• other pages have their own, separate TerrainPageContent instances, which will point to different heightmap files or sections of a big heightmap file
• loads/unloads heightmap data based on a GridPageStrategy
• the graphical display of this heightmap data is completely up to the SceneManager - in this sense the PageManager doesn't care what is actually done with this data.
• it is up to the TerrainPageContent implementation whether the heightmap data is kept around in memory for the duration of a loaded page. One possible implementation is to unload the heightmap data as soon as the page is finished being constructed by the SceneManager. How this works will depend on how the TerrainSceneManager is implemented, and whether we need to consult the heightmap data exactly once upon page-load, or more often.
• file saving does not need to be implemented, since terrain is static.

terrain physics support:

• option a) I use Ogre's stock TerrainPageContent class, which provides hooks to notify the application it has been loaded, so I can create physics.
• option b) I define my own TerrainPageContent subclass, which includes the physics code directly.
• Ogre3D supports one or both of these options.

2. stateless objects: Rocks, trees, and tower

• "near detail" objects: rocks and trees
• "far detail" object: tower
• implemented via 2 separate PageLOD instances.
• the far PageLOD instance includes only the tower
• the near PageLOD instance includes the tower along with the rocks and trees.
• the far PageLOD will be unloaded when the near PageLOD is loaded, which is why the tower must appear in both LODs.
• each LOD has their own ObjectPageContent instance.
• perhaps both ObjectPageContent instances reads the same file, but the far PageLOD only loads objects marked "FarLOD=true" into memory. The logic for how an ObjectPageContent retrieves it's data is either customizable or requires an application specific implementation.
• the paged data format may contain additional non-graphics properties, such as physics representation, which is processed by either an application specific ObjectPageContent subclass, or perhaps through notifications to the application.
• file saving for this ObjectPageContent (sub)class does not need to be implemented, since rocks, trees, and tower are static.

3. persistent state, unloadable objects: Gold ore

game requirements for ore:

• the gold ore can be mined for resources
• when ore is mined, is exhausted for 10 minutes, and may not be mined for additional resources during this time
• exhasted ore has a different appearance.
• there a lot of gold ore in the world, and I want it to be paged, both for memory optimization, and because it's simpler for the world editor to place the gold ore data in the same file along with the stateless objects.

implementation:

• option a) paging is implemented via a application-specific ObjectPageContent subclass, which must define a save routine to save and load persistent state to file (e.g. timestamp at which the ore was last mined).
• option b) we push persistent state out of the PageManager's responsibility. Gold ore state is tracked by application-specific code, perhaps mapping an ore ID to a last-used timestamp. In this case, the gold ore can likely be loaded and unloaded along with other stateless objects, and no save routine is required.
• in either option, the gold-ore data is not a simple mesh, but rather a game object with a collection of meshes corresponding to different mined states. This makes it different than trees and rocks, which directly feed meshes to the SceneManager. So my custom PageObject class would construct some these Ore game objects, which would then be responsible for telling the SceneManager which mesh to display.

4. persistent state, non-unloadable objects: Chatty NPCs

game requirements:

• the Chatty NPC must be loaded in memory at all times, as it constantly communicates to the player.

implementation

• option a: doesn't use paging at all. As pages corresponding to a chatty NPC's location are loaded, the application is informed, so that it can use it's own separate (non-paged) routines for inserting graphical data into the SceneManager.
• option b: uses paging, with a separate PageWorldSection and PageStrategy that loads upon game initialization, and never unloads. The reason for this curious "persistent page" is to be consistent with the loading APIs and file format implemented by the PageManager.

I should send money to anyone who read all that.

[SigfriedMcWild]

I like the idea of keeping the paging functionality separate from the the rendering functionality, as it isn't just a component of the rendering but a tool that the entire application can use to manage data (to me it seems a very similar situation to the input manager). In a similar vein I'd like to see the resource manager fully separated out in the same way.

Also jabberwoky I read through your entire post . It's seems a very good summary of what you should be able to do with the paging

[sinbad]

loads/unloads heightmap data based on a GridPageStrategy

Clarification - the GridPageStrategy would initially simply control the loading of the top-level Page definition. On loading this, the available LODs of the content of the page will be enumerated, and an initial LOD decision made by the LodStrategy instance(s) (I'm going to allow one strategy per content block here). It will be this that then requests the loading of the TerrainPageContent, which in turn loads the heightmap data. If no content LOD is used by the page then yes, the single set of PageContent will just be loaded.

Subtle difference, but I wanted it to be clear that the PageStrategy is only in charge of top-level Page instantiation / destruction decisions. Each page can then make finer-grained decisions about what content is loaded in at any one point, and then the chosen content loads itself. Of course once loaded, internal in-memory LOD may be a part of content types too (mesh LOD, terrain LOD), but this is opaque to the paging system since it doesn't involve loading new content (at least, not via the page system).

option a) I use Ogre's stock TerrainPageContent class, which provides hooks to notify the application it has been loaded, so I can create physics.

Yes, these kinds of hooks will be provided.

But yeah, your post reflects my thoughts on how this will work.

[Nauk]

Looks good and well thought out and I am definately looking forward to get my hands on it!

[sinbad]

Revised design diagram:

paging_design2.gif (23.25 KiB) Viewed 20620 times

The blue classes are examples of extension classes, which user code could provide alternative versions of.

I've also introduced a concept of PageContentCollection instead of PageContentLOD, since I figure there might be many reasons you would want a grouping / selection of content. There will be a LOD implementation but this can be replaced or supplemented with other types.

[cdleonard]

How exactly does this deal with complex object moving between pages?

Consider a complex object like a car which is a miniature "scene tree" by itself. It has one or more meshes but also a particle system for the exhaust and two spotlights attached as headlights (real fancy). These are all linked together by scene nodes. How do you move such an object from one page to the other without recreating it? Consider a racing game where you drive your car through several zones which are swapped in and out of memory as you pass through them.

The current approach of SceneManager to create custom objects derived from Ogre::Entity and Ogre::Particle system wouldn't work here. Not even creating custom per-Page SceneNodes would be allowed (since they're already used inside the car). I don't know if any scene manager actually uses custom classes for MovableObjects. Perhaps it would be better if core objects like Entity and SceneNode and SceneManager would not get derived outside of OgreMain.

Instead plugins could merely offer custom space partitioning schemes which arrange objects in the SceneManager for faster visibility queries. Since the partitioning schemes don't inherit from but merely aggregate existing objects it becomes rather easy to have our car stradle the boundary between two zones: it's referenced by the partitioning of both zones.

Is something like this planned for SceneManager 2.0?

[sinbad]

I tried to make clear above that the paging system is only concerned with paging in and out data. It need not exert any ownership on the content that is created as a response to that. I've already mentioned the deal with dynamic objects above, which applies irrespective of whether it's one object or a complex node tree of objects.

Regarding dynamic state, I think it's important to think of the page system as a manager of 'snapshots' of data. Once a page is loaded, it may well create dynamic objects, which will just be regular MovableObjects as far as Ogre is concerned. KungFooMasta's point about a boulder rolling across a page boundary is illustrative - unless the page in which the boulder starts was already loaded, the boulder could not roll into an active page because it doesn't exist yet. This is an unavoidable simplification and I think pretty easy to understand, and people will structure their paging to make this as least noticeable as possible - what's more difficult is unloading. If your page spawned a dynamic object and the page get's unloaded, what should happen to the dynamic object? Well, essentially this depends on how mobile the object is. If it moves a bit, but generally stays in the local region, then destroying it when the page is removed probably isn't a problem. If you can carry the object all the way across the world with you (e.g. you drive a vehicle that was spawned in at a page), that could be more of a problem!

I had earmarked ObjectPageContent as a placeholder for spawning general MovableObject instances in a page. My suggestion would be for the default behaviour to be to destroy the objects when their page is unloaded, but to have the ability to detach an object (or perhaps its parent node) from a page, at which point it is essentially immortal until you manually destroy it, just like a regular Movable. I could also allow an object to be re-attached to a page (e.g. when you put it down and walked away) so that when that page (might be a different one) gets unloaded, the object then gets destroyed. Of course, this would require the user to be aware of pages, but that's how I envisage it being possible.

Essentially each PageContent subclass is responsible for how it manages the objects it creates in response to paging activities. Those objects do not necessarily have to remain attached to the paging system forever.

In short, paging != unique scene partitioning. It's purely a way to trigger the loading and unloading of data. I have deliberately made the design assertion that there is no requirement for pages to be unique in space, or for content to remain attached to a page once it is instantiated. Each PageContent can deal with that how it wants - something like terrain will obviously have a much stronger ownership than dynamic objects. SceneManager 2.0 will introduce more 'ownership' issues because the intention there is to promote zone-local coordinate systems and PCZ-style features, at which point transferral of ownership becomes an issue - but that is not this feature. With paging, nothing changes as far as SceneManagers and object ownership is concerned - this is a crucial thing to understand. This is a stepping stone which adds paging features but changes nothing with regards to core scene management.

[Nauk]

I am not sure if this is the right thread / time to bring it up and I only understand probably not even half of the content & technology, but what I would wish for if the TerrainPageContent class is compatible or flexible enough to host current existing and well working terrain solutions such as TSM, ETM, ETL and the OverhangTSM and automatically provide stitching solutions for each of them - like a multipurpose stitching, skirting, blending class/manager - if that is possible in anyway. Another wish would be to expose the heightmap or meshdata of each terrain type publically in the SDK version of Ogre. Next thing that comes to my mind and would be awesome to have is a standard sample / demo shader / technology that provides UV contraction for slope texturing in heightmap based terrains to counter the texture stretching. But maybe that is already too content specific and goes too deep into detail for this thread - or would be relevant for the generic design for the paging engine already? *shrug*

[sinbad]

I am not sure if this is the right thread / time to bring it up and I only understand probably not even half of the content & technology, but what I would wish for if the TerrainPageContent class is compatible or flexible enough to host current existing and well working terrain solutions such as TSM, ETM, ETL and the OverhangTSM and automatically provide stitching solutions for each of them - like a multipurpose stitching, skirting, blending class/manager - if that is possible in anyway.

No - in fact, the terrain system I intend on introducing won't even use stitching to resolve page seams, it will use skirts instead. This results in far less geometry duplication and completely eliminates the problem of what to do at seams if you don't know about the page next to you, and effects of editing on stitching. On modern cards, geometry detail is generally high enough that skirts are fine, as shown by recent versions of SPT. Of course, anyone is free to implement their own PageContent class to wrap an existing solution, but I don't intend on providing this.

Another wish would be to expose the heightmap or meshdata of each terrain type publically in the SDK version of Ogre.

This data is specific to the PageContent - you'll be able to cast it to get at data like this if you want to, but I do not intend to promote terrain-specific interfaces to a higher level than that.

[Nauk]

Sounds good and castable is fine enough to be able to manipulate it with custom editor filters, brushes etc or to be able to create custom scene loaders working with the standard OgreSDK and without extra dependencies.

[DragonM]

Is there going to be a tool capable of massaging data into formats Ogre Paging can use? Mention has been made of third party support, but the planned features seem unusual enough that a native tool for converting plain old heightfields would be helpful, if only as a reference implementation.

DM

[sinbad]

Is there going to be a tool capable of massaging data into formats Ogre Paging can use? Mention has been made of third party support, but the planned features seem unusual enough that a native tool for converting plain old heightfields would be helpful, if only as a reference implementation.

DM

The intention is to make the format friendly to pulling data from existing image formats, as external files from the resource system if required. Obviously once in memory it would be saved out in our packed format but there's no reason why it can't be pulled in from images as is done with most systems now.

[xadhoom]

The intention is to make the format friendly to pulling data from existing image formats, as external files from the resource system if required. Obviously once in memory it would be saved out in our packed format but there's no reason why it can't be pulled in from images as is done with most systems now.

Do you mean heightmap images? This would be used in the very special cases of "not overhanging" terrains, wouldn´t it?
As far as I understand your notes the goal of your approach is a generic concept for paged data in general. Do you have an idea how generic paged data would/could be described outside of the code. *squinting to the new StreamSerializer class*

xad