Page 1 of 1

Terrain and Paging System Improvements for Very Large Worlds

Posted: Sun Mar 20, 2011 7:57 am
by Progor

My name is David Whittaker. I am working on my Ph.D. at the University of Alabama at Birmingham. I haven't completely nailed down my Ph.D. topic yet, but I am aiming in the direction of terrain generation techniques based on geologically accurate simulations. This is obviously not something I can do in a summer, but I think combining all the work and ideas I've generated so far to get me to this point into one project is. To begin my proposal, let me list some of the work I've done so far that has led me to this point:

displacementmapterrain: A looong time ago, one of my first OGRE projects was implementing displacementmapterrain - still in ogreaddons today. Its underlying idea - using a standard patch over and over again, only providing height data to each patch, and letting the vertex shader combine the height data with offset and scale information and the underlying patch prototype to greatly reduce graphics memory requirements is the underpinning of all my massive terrain work since.

Magna Terra: Fast forward to last year - this project was an over-specified, under-designed, and never implemented terrain engine for OGRE (and specifically, RoR). Many of the ideas of this project are applicable, but a more generalized terrain engine, for which these ideas could be one set of plugins, would be much more widely useful. So, the goal is to create an engine which could eventually accommodate all the ideas in this project, but not be limited to this particular implementation. I was a bit too focused on speed and optimization, and not on something that could be generally useful to the widest possible audience for this project. I will probably keep the OGRE project under this name.

MonkeyHorizons: This project for jMonkeyEngine represents my latest ideas, and most complete implementation. It has been partially released as a POC, but there is still work to do. So far, I have a system that can represent 8192 km^2 down to 1m resolution, with a single low-res bitmap to define the entire world down to 1km resolution, and a set of classes to generate noise to add procedural detail. The low-res bitmap can be as low-res as 2x2 (meaning the entire world is generated procedurally), or as high-res as the world is (meaning there is no procedural detail). There is still work to be done to make the texture splatting more realistic and controllable, but the geometry is working as intended. The new Magna Terra will begin as a port of this terrain engine to OGRE's paging and terrain concepts.

GSoC Proposal:


Paging System - Implement a PageStrategy which works on MonkeyHorizons' page replacement system - successively replace 1 page with 4 more detailed pages as they get closer to the camera, while merging 4 pages into 1 as the camera moves away. In essence, the current TerrainQuadTreeNode class maps to a single page in the new PageStrategy. Much of the Terrain code will be integrated directly into the paging strategy, removing the need for LOD determination at two different levels (one at the PageStrategy level to determine which pages to show, and again the same types of calculations within a page to determine which nodes to show). Another way of thinking of this system is as taking the quadtree approach within a single terrain page and extending it to the entire PagedWorldSection, rather than having two different concepts. The current MonkeyHorizons demo has a far clip plane set to 10,000km due to this approach, and is running at 100fps.

Terrain Generation system - this system is de-coupled from the paging or terrain system, to allow a variety of methods of generating terrain. A mix of reading raw heightmaps from disk and generating procedural content should be supported. The demo will include the North American landscape at 1024m resolution, with the Grand Canyon area down to 32m resolution, and perlin noise added in everywhere to add high-frequency detail down to 1m. The terrain generation system is also responsible for generating "climate maps" which are grayscale bitmaps at the same resolution as the height data which are used as input to the splatting system. These can be physical things like average temperature, rainfall, or water table depth, observed climates in the region, or procedural content like noise functions or calculations from surrounding height data.

Splatting System - this system is based on rules. Two GLSL or other shader language function will be required which maps from the terrain height, normal, latitude, longitude, and any climate maps which have been supplied to a set of textures and their percentages. Future work may include specifying these rules in a configuration file of some type and dynamically generating the shader functions. Either way, one of these functions will be called within the vertex shader to work with climate values which don't interpolate linearly (such as observed climate types at each location), and the other will be called in the fragment shader to work with interpolated values and choose the right textures and blending on a per-pixel basis. This system is still under development in MonkeyHorizons and may change somewhat by this Summer, but the basic idea is to provide a framework that will allow procedural engines to generate believable splattings, while leaving the possibility of fully defining the world and just reading the data in from disk, or even defining the world on a high-level basis - such as general climate information like rainfall amounts and average temperature - at very low resolution, adding some noise to make things interesting, and combining that data with the terrain information to make some believable worlds from low-res data.

Possible deliverables, probably future work:

Static object system - integrating the LOD system with the paging terrain system seems simple enough, since the current terrain system already does this, but the above deliverables already spell out a pretty full summer for a single developer, so unless this ends up being practically automatic if the other systems are designed well, it may be out of the scope of a single GSoC project. Either way, the idea would be to simply map the LOD of all the objects on a page to the LOD of the page itself, and probably combine many objects into one on pages far from the camera.

Flora System - extend the static object system to work with tons of objects, fading out grass at a certain distance and bushes at a farther distance, and probably even replacing an entire forest of trees with a single texture at a certain distance. It would be great to integrate ngPlant or a similar software to create a fully open source replacement for SpeedTree.

World Editor - Add heightmaps at different resolutions over different parts of the world, add layers of noise to add detail, paint noise amplitudes over the terrain, paint climate maps and edit noise function parameters to generate detailed climate maps. All the editing should be done in real-time on a live instance of the world, and data should be saved as needed before unloading edited data (along with undo information). Finishing this is obviously out of the scope of a summer project, but it may be worthwhile to get a basic editor in place to make the engine more immediately accessible.

So, what do you think of the idea, and where should I draw the line for the deliverables? I'm usually too ambitious for my own good, so I drew the line earlier than I would have liked, but I'd rather go above and beyond my commitments within the alloted timeframe than not make them because I aimed for Mars and hit the moon.


Re: Terrain and Paging System Improvements for Very Large Wo

Posted: Sun Mar 20, 2011 8:17 am
by stealth977
IMO most important part is having a Seamlessly Paging Terrain Component, all the rest like procedural generation, splatting, plants etc. are only viable once the first requirement is met...

And about world editing tool to test/demonstrate features, well, we could add it as a plugin to Ogitor, which would be much faster than coding such a tool from scratch...

Re: Terrain and Paging System Improvements for Very Large Wo

Posted: Sun Mar 20, 2011 8:41 am
by tuan kuranes
The three points here seems a bit borderline here, as they all could be implemented in the application instead of ogre core due to huge modularity of the ogre terrain system, therefore more demo/application/editor side than Ogrer core side.

Usually, gsoc focus on ogre core, here terrain system, and as a results gives demos, application feature.

ie: in your case "page-relative co-ordinate systems", "hierarchical paging", "Page lod entity load" demo would be a better scope imho.

Btw, splatting is really a bit old technique here, Virtual Texturing is really the way to go nowadays, so much advantages...

Re: Terrain and Paging System Improvements for Very Large Wo

Posted: Sun Mar 20, 2011 10:55 pm
by Progor
Thanks for the comments so far.

Re: Splatting vs. Virtual Texturing

Virtual Texturing is an awesome technology, but I am wary about using it for this terrain for this reason: the absolute maximum size of the virtual texture is stuck at 8192k x 8192k. You only get 23 bits of mantissa in a float, after all, so you simply can't index any more texels than that. Now, let's say you max out the world size with an 8192km x 8192km world. This leaves you with a minimum texel size of 1m x 1m. Obviously, this is barely detailed enough for a flight simulator, and simply won't work for any application that ever has you walking around on this terrain. So, let's say you cut down the world size to 64km x 64km, a reasonably large world for your average MMORPG, but by no means huge. This is the size of the recently released game Earthrise, for instance. Even then, you end up with 128 texels per meter, or a texel size of just under a centimeter. This seems to still be a little on the low-detail side for modern games. 16km x 16km gets you down to a 2mm texel, which finally seems reasonably sharp, but a 16km^2 world is pretty small by MMORPG standards. So, you'd have an arbitrary level somewhere down the quadtree that each terrain patch gets its own megatexture, and above that level all terrain patches use a single megatexture for the whole terrain... I guess that works, but I was trying to go for a system where the whole terrain system is self-similar at all levels. Splatting only requires a texture the size of or a few times larger than the terrain size, so it seemed to be a more natural fit for really large terrains. In the end, I guess the texturing system for the terrain should be extensible enough to allow either approach.

Re: application vs. core

I think I see what you are saying here. I may be on the right track for a GSoC project, but I need to word my proposal in terms of Core OGRE components rather than application features. In the end, I'll end up doing much of the same work, but the application features I've listed are the demo that uses the core features that I need to focus on in my proposal. I'll update the wording of the proposal in terms of the core features you mentioned tonight and see if it stands up as a more solid GSoC project. By the way, the coordinate system currently in my current demo is a camera-centric coordinate system, where the pages are located at global coordinates (I get away with this because the pages are all located at integer positions), but this will have to change to precisely place objects on pages, so a page-relative coordinate system is certainly in scope. Rather than a page ID and location within that page though, to reinforce the "infinite depth" of the hierarchical paging system, global coordinates defined as macro integer position and float offsets from that integer position may be more applicable.

@stealth977: I agree with you, a seamless terrain system is the fundamental component that has to be in place before anything else to be worth including. That's the point of the hierarchical page system that it is all built on top of. You get a huge multi-resolution paged world that has no zones at all, and the generation system is completely decoupled from the paging system, much like OGRE's paging system is decoupled from the content it is paging in. So, the core functionality that I want to develop is the hierarchical paging system that loads in different LODs of objects based on the LOD of the page... which is based on the distance from camera, so everything works out. Whether it's loading in terrain, static objects or otherwise on each page is secondary to providing the mechanism for loading the world seamlessly.

An ogitor plugin sounds like exactly the direction I want to go with the editor.


Re: Terrain and Paging System Improvements for Very Large Wo

Posted: Mon Mar 21, 2011 8:30 am
by tuan kuranes
Re: Splatting vs. Virtual Texturing

Anyway the hierarchical paging must keep its independence and allow each page its strategy, as you should be able to do geomorphing terrain for near pages and ray-cast terrain for distant pages, satellite texture for distant pages, and splatting or virtual texture for near pages, etc.

But I don't quite get your point about, no per hierarchical page "texture strategy" ?
You would "hierarchical page" all terrain data but texture ?
Texture is the heaviest, slowest and the real minimizing factor and more important to visual quality ?
Afaik, MMORPG have texture pages and tiles, so does flight simulator.
Splatting has the same plague as virtual texture considering a universal texturing solution, as it's "splat tile" also texture size limitation, and therefore depending on camera height, you won't use the same texture tiles anyway ?

Re: application vs. core : exactly my point.