Future plans for Ogre::Terrain?

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
Post Reply
User avatar
stealth977
Gnoll
Posts: 638
Joined: Mon Dec 15, 2008 6:14 pm
Location: Istanbul, Turkey
x 42

Future plans for Ogre::Terrain?

Post by stealth977 »

It's been quite some time i am playing with Ogre::Terrain/TerrainGroup due to my work on Ogitor. We started integrating it to Ogitor since the very first days and had quite a lot of experience with it during its progress, so now i would like to share my doubts about its future:

Ogre::Terrain has evolved rapidly and became a much much better terrain renderer compared to old TSM or ETM, it is quite stable and feature rich, also quite editor friendly (still some needed functionality is missing but, i know its W.I.P). So that you can use it anywhere you would use TSM for and get much better results both in terms of visual quality and performance, but;

Everything was OK till we integrated PAGING in Ogitor. A single Ogre::Terrain is quite performant up to sizes of 2049x2049, but as soon as you want to use a larger terrain, divided in to pages, the problems start to show up:

1 - Any page size above 257x257 causes visible stalls (whether 200ms or more, but visible) on regular machines

2 - Memory usage is huge, if you set paging hold/load radiuses to load and hold just 1 page around the page you are currently in, it means 9 pages loaded all the time, assuming pages are 513x513 (a very much used page size), the memory usage jumps up to anywhere between 400-700 MB, and thats when i frequently use freeTemporaryResources!!!

3 - The data file size is huge, 10MB per page with 513x513 map/blendmap/lightmap (makes 10bytes per vertex/cell, but actual data is around 40bytes per cell???) which in turn makes it real hard to deploy large landscapes 2049x2049 4kmx4km landscape takes around 160MB...

4 - Paging is not adequate, think of an island using 4x4 pages, everywhere else is covered with water, you see the page you are on and the neighbour pages, then you see water in the distance, you start to get close to water, suddenly the next page loads and where you think there is water, appears landscape (a huge visible pop), of course you cant increase loadRadius, since it would mean lots of pages in memory which we cant afford, Ogre::Terrain definetly needs some way to load lower resolution maps at the distance and improve them as that page gets closer, also the far away pages should consume less memory, so that we can increase the loadRadius without multiplying our memory usage or paging has no value...

5 - Just a little suggestion: Lightmap and Colourmap can be combined together as an optimization (2 distinct memory copies, but one final GPU texture), their affect on the shader is same, of course it can be done by writing our own MaterialGenerator

6 - LERPing between layers is nice and prevents over/under saturation, but GPUs have very effective scalar multiply+add instructions which are handy when the form final colour = layer1.influence * layer1.colour + layer2.influence * layer2.colour ........layerN.influence * layerN.colour is used. In one instruction, 4 layers can be combined...Again of course it can be achieved using custom materialgenerator...

7 - There is a problem of Min/Bax batch sizes, if 9 - 33 is used for 513x+ page, it consumes all memory and crashes???

The most interesting part of my experience was, i can synchronously load 100 entities in a page in much less time than loading a pre-calculated 513x513 map. Of course those entities sharing a few meshes speeds up the process, but still, somehow page loading is slow, of course if loading pages at different resolutions depending on the distance would help a lot in this case...

Anyway, at current state of the code, it doesnt seem feasible to use Ogre::Terrain/TerrainGroup as a paging landscape (it may seem to work on high-end developer machines, but not on average user machines)...

I hope it can be imroved to be feasible for such projects too... (as a measure, whole landscape should be able to be viewed from a high enough distance, if i have 16 pages, i should be able to view them from a high enough distance WITHOUT the TOTAL COST of FULL RESOLUTION TERRAIN)

Ismail TARIM
Ismail TARIM
Ogitor - Ogre Scene Editor
WWW:http://www.ogitor.org
Repository: https://bitbucket.org/ogitor
User avatar
spookyboo
Silver Sponsor
Silver Sponsor
Posts: 1141
Joined: Tue Jul 06, 2004 5:57 am
x 151
Contact:

Re: Future plans for Ogre::Terrain?

Post by spookyboo »

Maybe some kind of impostering could be used instead of real terrain.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: Future plans for Ogre::Terrain?

Post by sinbad »

I think I already made it quite clear in the Showcase thread that I wasn't happy with the speed of the terrain paging yet, that's why it's not enabled in the demo and why I'm not shouting about it despite it actually being there. I hope to be able to work on this some more in future to make it faster & more efficient. Right now, I'm only suggesting it's production ready as a simple non-paged group although clearly the paging implementation is there too. I've already talked about things such as:
  • Vertex compression
  • Partitioning the geometry into different LODs for loading as well as displaying, since the hierarchical structure already allows for that
  • Compressed / encoded data options in addition to raw float data
I hope to have time in future to implement these things, or maybe others will help or a GSoC project will involve it. Remember, this is all volunteer time and the existing terrain work represents literally hundreds of hours of work already. Sorry the paging aspect is not perfect yet, but that's life (and open source); if I had another couple of months to work on it I'd gladly invest them but I had to move on, at least for now. It's on my wishlist.

I'll just have to disagree with you on whether a Photoshop-style LERP or multiply-add is best; yes I know the multiply-add is cheaper but it has its own issues too. You can change that with a material generator anyway, as you say.

I kept the lightmap separate from the colourmap because it's faster to update that way, and its also faster if other renders just want to reference the lightmap.

Please just raise bugs on other issues such as crashing under certain circumstances.
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: Future plans for Ogre::Terrain?

Post by CABAListic »

As for the LERP vs multiply-add: I have plans to implement a separate material generator which does the latter for my personal project. I can't give an exact time schedule, but it's on the top of my list now. Can't say if it will end up in Ogre as a core alternative, but if not I'd gladly share it with you, stealth (that is, if you haven't implemented your own solution by then).
User avatar
stealth977
Gnoll
Posts: 638
Joined: Mon Dec 15, 2008 6:14 pm
Location: Istanbul, Turkey
x 42

Re: Future plans for Ogre::Terrain?

Post by stealth977 »

Ok, now since Sinbad is retiring/retired due to physiological reasons:

What will happen to Ogre::Terrain/Ogre::TerrainGroup and Ogre::Paging?

Yes i know and can guess that sinbad will continue to contribute to the project, but ONLY developer of those components being retired, who will be in charge of them from now on?

There is a lot to be done for those components:
- Terrain components still need improvements in terms of stepped loading of the heightfield, or Heightfield LODS (distance terrains should start with a coarse heightfield and increase detail as it gets closer to viewer) to improve paging, currently heightfields of size 513x and above cause serious paging lags

- Paging should be extended to provide a default loader for general ogre objects...

Etc. Etc...

I hope, those components can have a Maintainer...
Ismail TARIM
Ogitor - Ogre Scene Editor
WWW:http://www.ogitor.org
Repository: https://bitbucket.org/ogitor
CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
x 58
Contact:

Re: Future plans for Ogre::Terrain?

Post by CABAListic »

sinbad retired as the *lead* developer for Ogre, not as a developer ;)
If in doubt, though, the Terrain component will work like any other area of Ogre: It will be improved whenever someone has the need and time to do it.
BTolputt
Greenskin
Posts: 121
Joined: Thu Feb 18, 2010 8:05 am
x 2

Re: Future plans for Ogre::Terrain?

Post by BTolputt »

That's a bit of a non-answer there Cabalistic ;)

Yes, sinbad is retiring from lead developer and has mentioned that he'll be spending less time in general on Ogre due to his health. This, in real-terms, suggests that any improvements/bug-fixes on his terrain code will be slower than previous history would suggest.

In any case, the only person who can honestly answer that is sinbad himself and he'll be busy with preparations for his stepping down in what free time he has in the short term.
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: Future plans for Ogre::Terrain?

Post by sinbad »

I'll still be around contributing code, and terrain is one area I'm most likely to do so. I already added GPU vertex compression and deflate disk compression on the terrain data. I'd also like to tackle more material fallbacks, the hi/lo data load split and a couple of other things. I can't say precisely how much time I'll have, but they're on my list.

Implementing paging of other objects (e.g. Entities) is something anyone can do, it doesn't need to be me. The framework is all there, it just needs more thing 'hanging' on it.

Obviously the fundamental principle in open source is that each person gives the time that they can, in areas they're most interested in, and I'm no different now. I happen to still be interested in the terrain so I'll most likely still be adding to it. But that doesn't mean everyone else should just sit back waiting for me to do so - that model is called 'traditional proprietary development' ;)
User avatar
Praetor
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 3335
Joined: Tue Jun 21, 2005 8:26 pm
Location: Rochester, New York, US
x 3
Contact:

Re: Future plans for Ogre::Terrain?

Post by Praetor »

I'm sure people will slowly start to hop onto the terrain to contribute. Steve moved very fast developing it so he put a lot of code together without anyone else keeping up. As we all get used to it and use it more (like stealth977 with Ogitor) the rest of the community will start to contribute more.

On the topic of the terrain I might as well toss out one little issue of my own. With a "walking" style camera (locked to the terrain, low viewing angles) I noticed something interesting. Most of the terrain was very fast and the LOD worked really well. However, looking in one direction I got a noticeable FPS drop (~200 down to ~80). I went into wireframe and noticed that even distant patches in that direction were using a lot of polys. I suspected it had to do with the general shape of the terrain, which was just right that from the low-angle camera. The solution was to tweak the terrain shape. However, if I'm correct in my assumptions I wonder if there's a way to override the existing LOD algorithm with a straight distance calculation. Basically a switch that says "forget the normal LOD factor, that patch is too far use this low LOD, please."
Game Development, Engine Development, Porting
http://www.darkwindmedia.com
User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19269
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 66
Contact:

Re: Future plans for Ogre::Terrain?

Post by sinbad »

The LOD factors are calculated on a pretty standard 'worst case scenario' height difference basis, a particularly bad case in one quad node can obviously 'infect' the rest of the node and push its LOD upwards. The main downside of the hierarchical batching is that aberrations like this affect a wider and wider area of the terrain the further up the quad tree you go - that's the nature of grouping things together of course. If you wanted, you could tune the values in the quadtree by hand instead of accepting the calculated values, although you'd have to remember that your alterations would be overwritten if the terrain was edited (which triggers a delta recalculation).
Post Reply