Simple Paged Terrain [v2.00 Released! Page 21]

HexiDave · Post by **HexiDave** » Fri May 11, 2007 9:50 am

I'm listening to a political debate, so excuse me if I write something stupid...

Different stages of "chunk" size is already possible in several ways:

- Streaming data in from the hard drive (memory mapped file reading) to get "coarse" data or by synthesizing "coarse" data is already possible and it will fill in the gaps in data when you request. Here's an example:

My Pages are 513x513 vertices and (I'm going to skip the bit about normals as it's all done passively) have heightmaps to match. Instead of synthesizing 513x513 values for every page as you load it to build 8x8 Tiles at 65x65 vertices each, I can use a pyramid of sectors governing the setup for Tiles of a Page. Lets see if I can explain that... Imagine looking down on a full Page - you see 8x8 Tiles - you take a corner of 4x4 and take it to a coarser level of the pyramid of Tile LOD scales by instead building 2x2 Tiles that take up the same physical size, but much less data needed to build it and requires less geometry to render. I can build a Page at 8x8 Tiles or a single big 65x65 Tile needing only 65x65 total height values.

Now, the cool thing about that is also that you can hook those tiles together with LOD (since they use the EXACT same GeoMipMapping IndexBuffers as they would at full detail) and pull data from them at that level of the pyramid without an issue. As you request detail, it'll either pull it from the hard drive (or you can stream the whole thing in at first and just use the LOD pyramid) or synthesize it for ONLY the height values that the operation requires - it won't build all Level 3 pyramid data at once, it'll build it as it's requested and fill in the heightmap and/or Vector3 cache that way.

Ok there was a lot to that example... moving on...

- You can simply set the general system to use, say, 513x513 heightmaps with matching Page/Tile geometry, but you can also specify certain areas (flat-lands, deserts, etc) to use very low resolution geometry like 65x65 or 129x129 with matching heightmaps and the LOD will STILL work due to the nature of the stitching algorithm.

- You can use manually created meshes (which I'll provide an OBJ exporter for, so you can take the created geometry from the terrain into Maya or Max or something) and the tiles surrounding it will stay at the LOD that would be required in order to maintain proper edge (i.e. your terrain Tile was exporter at LOD 0 - all the tiles around it will treat it as LOD 0 and they themselves will remain at LOD 0 such that no gaps appear on the edge of the mesh.)

- This won't be in right away, but it's do-able: you don't have to build normals until you're nearer to the terrain Page/Tiles. Extremely far away mountains might not really need normals if they're shrouded in fog, thus you'd build normals/tangents in when they're needed.

- The last option I really have planned to implement would be what I believe is called "Terrain sprites." Basically I'd create 1-2 levels of huge billboards or sky-cubes that would represent far-distant mountains by rendering them to a texture at full resolution and then hiding or even deleting the geometry in favor of a fast 2D texture as a background.

Ok that's a lot of stuff covered there, but it's all well within reason. I had a lot of practice dealing with funky LOD schemes with my OgrePlanet project, so I know I can do all that stuff about the LOD pyramid without major problems and it'll all be customizable (on or off, how many levels of the pyramid to let it scale into, which levels to use (i.e. use only 0, 2 and 4) and determining the maximum resolution for a Page (i.e. all are normally 513x513 but this one will be 129x129)).

I'm going to get the core features in first, then I'll do all the fun LOD things to make it sparkle.

Azatoth · Post by **Azatoth** » Fri May 11, 2007 9:56 am

I'm thinking of what shown in this screenshot: http://tulrich.com/geekstuff/images/puget-1-wire.jpg

I.e. LOD that adapts itself to the amount of detail needed.

The screenshot is from http://www.vterrain.org/LOD/Implementations/ .
Perhaps an even better approach is to make the LOD generation decoupled with an easy to implement interface, allowing for different LOD mechanisms to be plugged in? There seems to already be a lot of different libraries which does terrain LOD; if those could be reused in Ogre it would be excellent.

HexiDave · Post by **HexiDave** » Fri May 11, 2007 10:02 am

Yep, I had intended to create a Continuous LOD system for OgrePlanet after I got the whole thing squared away, so it's certainly do-able.

De-coupling the LOD mechanism is easy the way I have it setup, so if you just want to brute force it or use GeoMipMapping or Continuous LOD or Chunked LOD or Clipmapping or whatever, you can substitute it either manually or (if I get my ass in gear later - you'll probably have to bust my chops on this one...) writing it as a big compile-time mechanism using #defines and such.

Honestly, the only reason I haven't plugged Continuous LOD in to my implementations so far is that I've never made my own version before. Almost all of the setups I've seen skip using index buffers and just manually setup draw calls to render a newly batched pile of triangles. If someone else comes up with even a basic Ogre, buffer-friendly version of it, I'll put it in and put your name all over it if you like (can't do anything better than that, I'm afraid

)

For now, though, it's going to sit at the back of the bus as I've got a good sized work-load of features to put in first and de-coupling my current LOD system won't be hard.

Post by **Jabberwocky** » Fri May 11, 2007 8:41 pm

HexiDave wrote:Anyone have any features they haven't seen mentioned so far that they'd like in this thing? So far most of the features I already had planned were mentioned and are easily do-able, so adding more to the list isn't going to hurt much

I feel I have a good understanding of how the terrain geometry is going to work. What do you have planned for the texturing side? Is some form of alpha splatting available for combining textures? Is it procedural or will it use an alpha-map? Will it allow for a custom material file like TSM (which pretty much allows you to do whatever you want)?

HexiDave · Post by **HexiDave** » Fri May 11, 2007 8:52 pm

For now you can set your own custom material/textures and put them in and it accepts them just fine - you can do this by Page (i.e. pPage->getMaterial()) or set it by Tile. If you set a Page's material after the Tiles have been built, you can use a little filter command that will reset all the Materials for the Tiles to that material (such that you don't have to go through each Tile by hand and set them to the new material).

I'll set up a system that lets you put your own textures into a list for alpha-splatting (although this should really be done in a shader, methinks) either as single alpha maps or a 4-channel RGBA map where each channel counts as Alpha. Currently I have an RGBA map that's created when the Page is with RGB being the normal map and alpha channel being the heightmap (so you can do height/slope texture splatting).

And a quick update before I run off to eat: I figured out a new method for doing the wrap-around code, so I have to clean up a few things, but it works a lot better than before and helps a lot with precision issues. I have to work on a few "special case" things for the edges, such that normals and vertices are correct along the wrap around point, but do it in such a way that it doesn't load a lot of extra garbage (that is, when Page 65535 (or whatever your max Page is) meets Page 0, they don't line up with standard noise functions, so I'm just going to linearly interpolate from Page 65535's Heightmap into the reversed Page 0's Heightmap - this way the edges will meet, normals will be correct and it won't be an ugly transition.)

Post by **CABAListic** » Fri May 11, 2007 9:02 pm

For (editable) alpha splatting, you could also pull in the SplattingManager from my Editable Terrain Manager (as soon as the next version is finished). It's independent of any specific terrain rendering since it just manages the splatting textures. So you could use it and build a custom material from that for any of the tiles.

Post by **jacmoe** » Fri May 11, 2007 9:07 pm

Collaboration between you two would be awesome!

HexiDave · Post by **HexiDave** » Fri May 11, 2007 10:00 pm

I wouldn't mind a collaboration, but for now I've got a pretty solid battle plan for my engine, so I'm not sure what I could offer you for your project or what I'd need help on.

As for alpha splatting, I'm just storing, per Page, a std::vector<> of all the textures it will require and then it'll feed them to the material as needed, then destroy them when it's time for them to go. Since the Pages are fairly self-sufficient, it can either synthesize any of the maps (alpha, normals, height, etc) or load them from the disk (I could combine 4 Alpha splat textures into 1 RGBA texture for the shader to use if they're 8-bit splat maps) and it wouldn't really have any bearing on other systems.

And, of course, since the project is open to suggestions, people can either build something they like into it or let me know what they want done and I can see if it's easy to fit into the system somewhere. So far the ideas put forward are good and well within reason to put into the system, and I've got them all noted and have a place for them.

Though, if you check the source code when I start making more releases, you can tell me if I'm doing something stupid and we can hammer it out

On a completely unrelated side-note, anyone else with Firefox have a problem posting on here where it crashes after you hit Enter? I don't seem to get it anywhere else but here...

Post by **CABAListic** » Fri May 11, 2007 10:08 pm

There's actually nothing you'd need to do from your side except provide the capability to plug in custom materials (but you have that already). My SplattingManager does nothing more than handle the splatting maps dynamically in memory and provide means to edit them. They can be referenced by name and thus be used in any material in any form wanted.

Of course, it is not aware of paging. Unless one would want to use a fixed texture for the whole terrain (which may easily not be enough to make splatting detailed enough). So on the user side one would probably have to build a manager on top of the splatting manager which uses multiple splatting managers and materials (one for each page) and creates or destroys them as pages are loaded/unloaded.

Lord LoriK · Post by **Lord LoriK** » Sat May 12, 2007 9:04 pm

HexiDave, have you thought of asking for a place in Ogre Addons? I think this does deserve it.

Edit: after peeking a bit the code, I've found a few things that might make it cleaner:

* Create a namespace to put your classes.
* Replace those preprocessor symbols with constants inside that namespace.
* Don't include Ogre.h, because it includes TONS of headers you don't need and makes the compiling process too slow. Instead, include only those meaningful headers.

Hope you don't mind the suggestion, but I think it's easier to do while the codebase is still young and small.

HexiDave · Post by **HexiDave** » Sun May 13, 2007 12:24 am

I agree entirely with those suggestions - they're "to-do" things as right now I'm modifying them so often I don't feel like dealing with type-changes and GlobalSettings::blahblahblah everywhere. Also the Ogre.h inclusion is another example of pure laziness on my part. When I get everything settled, I'll go back and make everything secure and faster for compile.

I actually got screwed over this weekend - I'm doing a lot of non-Ogre related work that I hadn't intended, so a lot of stuff I wanted to work on got pushed back a few days. The wrap-around system I was planning didn't work the first time, so I have to sort out what I did wrong and re-do it. I think I know what I did wrong, but I have to sit down with a pencil and paper a bit

Everything else is in good shape, so I'm taking my time with this to get it right.

As for having this as an Ogre Addon - that can wait until I get it filled out with features and user-friendly. The current 2 code releases (as you noticed) were far from Addon material

So, I promise I didn't forget about posting code and such, but I'm just backed up a bit with surprise work this weekend. When I get the wrap-around system done, I'll post revised code.

Avihay · Post by **Avihay** » Sun May 13, 2007 5:30 pm

First thing first: WOW!

HexiDave wrote: Anyone have any features they haven't seen mentioned so far that they'd like in this thing?

* Loading parts of the terrain from a Topographic map like data (closed curved lines that represent points that are at the same hight. usualy the lines are arranged in jumps of 10 or 100 meaters/feet)
I'm guessing your anser to that will be to make a loading plugin to do all the work to convert it into a hightmap, but I still like to see if you have better ideas

HexiDave wrote:So far most of the features I already had planned were mentioned and are easily do-able, so adding more to the list isn't going to hurt much

/me tring to prove you wrong:

* Making the terrain deformable.
I don't expect to be able to do it in real time, but then again, you can blind the user from the deformation are for a few frames so there won't see the poping (explotion particels for crash craters, smoke for volcanos and the like)...

HexiDave · Post by **HexiDave** » Sun May 13, 2007 8:51 pm

Pretty sure I stated early on that the terrain was going to be editable and customizable in real-time. If I didn't say it here, then I think I said it in another thread...

So, yes, it's already on my to-do list.

As for topographical maps... how the hell are they read? I didn't read that whole thing, but do you have a file format there or is it just scanning the curves or something? I never really dealt with topographical data types, I was always more on the fractal side of things.

Post by **jacmoe** » Sun May 13, 2007 8:59 pm

AFAIK, they are most often in DEM(Digital elevation model) format, or in DRG(Digital raster graphic) format.

Post by **jacmoe** » Sun May 13, 2007 9:02 pm

The most common DEM is the USGS DEM.
It's not only used for real-world data, but other projects are using it as well.
IIRC, the Middle Earth project is set out to create DEMs of the Tolkien worlds..

HexiDave · Post by **HexiDave** » Sun May 13, 2007 9:50 pm

Ah, well if it's a recognized format, then I can probably do something with it.

EvilPickles · Post by **EvilPickles** » Sun May 13, 2007 10:56 pm

Lol, raicuandi had the right idea. Infiniti -- love the name!

HexiDave · Post by **HexiDave** » Sun May 13, 2007 11:36 pm

EvilPickles wrote:Lol, raicuandi had the right idea. Infiniti -- love the name!

Not sure I know what you're talking about... Did he reply to me in another thread somewhere about this?

You've broken my brain

Post by **Falagard** » Mon May 14, 2007 2:50 am

Different stages of "chunk" size is already possible in several ways:

Clarify - it's already possible with the current code and I'll be able to look at your code and figure it out ... or you have to do additional work before it's ready.

HexiDave · Post by **HexiDave** » Mon May 14, 2007 3:25 am

It already works in a "dummy" version of the code - I haven't integrated it yet - I want to work out this wrap-around system as the basis will give me the ability to avoid precision issues in the basic setup. I'm going to test my new system out tonight (free from all the crap I've had to do this weekend, finally) and see if it works.

Basically it figures out what Page your over and uses that as a basis for where new Pages are placed (i.e. camera is over Page 10,10 - the page to the left is 9,10, but the build system will see Page 0,0 and Page -1,0 to do the construction) as well as movement of old Pages (the reason this was coming up with weird errors was because I wasn't storing the right information for the calculations for movement).

The basic part of that works (the precision part), it was just the wrap-around system going funky on me.

Once the wrap-around system is completed I'm going to spend some time cleaning up my code and refactoring it a bit. I'll try to add in a user-configurable system so people can actually start using it instead of saying "ooh pretty" - not that I don't like hearing that - I figure letting people actually use it would be good.

Bobf · Post by **Bobf** » Tue May 15, 2007 1:07 pm

Hows this goin' HexiDave?

Most interested in hearing updates. How are you planning to release this anyway? As a plugin? a bit of code? dll?
New screens?

PLSM2 is sorta playing up for me, so i'm still nosing around for alternatives.

Azatoth · Post by **Azatoth** » Tue May 15, 2007 1:13 pm

HexiDave wrote:Ah, well if it's a recognized format, then I can probably do something with it.

Perhaps this is something you could use? http://www.gdal.org/

kneeride · Post by **kneeride** » Tue May 15, 2007 3:19 pm

Ha! I've always wanted to design my terrain using contour lines. I even considered making my own editor once.

HexiDave · Post by **HexiDave** » Tue May 15, 2007 8:26 pm

Sorry about delays - I got screwed over again and I have to do some unexpected work.

The wrap-around system works except that the auto-loader system doesn't want to play nice with it. I've almost got that fixed and then it's good to go. As it is now, there are no precision issues at all, so your really huge terrains won't get screwy on you.

I promise I haven't given up or forgotten anything, just getting piles of extra work and it's slowing me down. I should have a new release for the weekend.

HexiDave · Post by **HexiDave** » Thu May 17, 2007 5:23 am

I am currently SHAKING MY CALCULATOR IN FURY!

Why you ask?

0 & 3 = 0
1 & 3 = 1
2 & 3 = 2
3 & 3 = 3
4 & 3 = 0

0 & 2 = 0
1 & 2 = 0

.. WHAT?

Ok, I program often but I never explored the subtleties of some bitwise procedures. Namely, using AND ( & ) on an even number results in non-wrap-friendly behavior.

Not only did I not know this, but I just now know this after fighting my 3x3 page simulation (that is, Pages numbered 0 through 2) for a few days. All those times I failed to get my code to work, all that time I spent coding and getting weird results... I finally perked up on the fact that I wasn't displaying my wrap-around page results (the math checked out with my 2x2 Page simulation from way before, but "that is too small", I said to myself...). Within 30 seconds of displaying the wrap around results on my debug window, I wanted to head-butt my calculator.

So, in short: Wrap around works now! Yay! I just have some tweaking and cleaning up to do (soooo much code wasted *cries*).

Give me a day or two to put some new stuff in and clean up the giant mess and I'll even try to put up an SVN repository (never used one, so beware) of the code so people can see the horror that is undocumented code!

Anyways, sorry again for the delays (and more to the point my absolute stupidity) - you'll get something to play with before the weekend.

Ogre Forums

Simple Paged Terrain [v2.00 Released! Page 21]

Texturing?