[GSoC 2012 - Accepted] Improve and Demo the Terrain System

Threads related to Google Summer of Code
Post Reply
User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by masterfalcon » Thu May 31, 2012 5:27 am

My only concern is the possibility of performance hiccups when the LOD changes.

Perhaps we could attempt to predict when LOD will change and load on the fly?
0 x

User avatar
duststorm
Minaton
Posts: 921
Joined: Sat Jul 31, 2010 6:29 pm
Location: Belgium
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by duststorm » Thu May 31, 2012 5:24 pm

Are there no such strategies already implemented in other engines, that predict the next pages to load and stream them in before they are actually necessary?
If you are able to allocate enough RAM space to load all neighbouring pages in, the camera can move to any neighbour, then the only assumption you make is that the camera will not move faster to another page than that pages can be loaded from disk. I believe this is the sort of assumption that is made in some large outdoor rendering setups. (Just Cause 2 comes to mind: the only loading you ever see there is if you teleport from one area to the other. When traveling through the world regularly you never notice any loading, as the necessary data is streamed in memory just before it is needed.)
0 x
Developer @ MakeHuman.org

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Sat Jun 02, 2012 3:22 am

masterfalcon wrote:My only concern is the possibility of performance hiccups when the LOD changes.
Perhaps we could attempt to predict when LOD will change and load on the fly?
Ehh, the topic reply notification seems broken.

Performance hiccups is indeed one of the problems, which you can notice by flying around fast. So I want to reduce the loading-from-disk step when the LOD changes. As I mentioned above, this step doesn't save any memories, as well as performance.

We can predict that one LOD seems to be needed by watching the DeltaData. See:
...
|
| <-- The biggest delta data is noticeable!
| <-- The LOD seems to be needed?
|
...
For this reason, the whole copy of delta data should be always available, too.
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

User avatar
aguru
Goblin
Posts: 236
Joined: Tue Feb 26, 2008 5:48 pm

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by aguru » Sat Jun 02, 2012 8:57 am

Quite unrelated to the current discussion, but I wondered if you are looking for any beautiful heightmap data for the showcase demo? We have some experience with those.
http://www.ogre3d.org/forums/viewtopic. ... 25#p368126
0 x

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Sat Jun 02, 2012 9:44 am

aguru wrote:Quite unrelated to the current discussion, but I wondered if you are looking for any beautiful heightmap data for the showcase demo? We have some experience with those.
http://www.ogre3d.org/forums/viewtopic. ... 25#p368126
Thank you so much! The island is very beautiful!
In my showcase demo, heightmap will be generated randomly. I hope it look good :D
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

User avatar
aguru
Goblin
Posts: 236
Joined: Tue Feb 26, 2008 5:48 pm

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by aguru » Sat Jun 02, 2012 10:21 am

Yes, generating it randomly is a good idea, that way distribution size won't get too large ;)
0 x

User avatar
duststorm
Minaton
Posts: 921
Joined: Sat Jul 31, 2010 6:29 pm
Location: Belgium
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by duststorm » Sat Jun 02, 2012 12:59 pm

Prodecural techniques can also be very handy to avoid having to cache large amounts of data in memory or read it from disk.
Not entirely sure whether this can apply to terrain, as it would probably look weird if it is different every time you return to the same area. But for terrain textures and details like vegetation placement it could make a big difference if you could just randomly generate it instead of having to load it from data.
0 x
Developer @ MakeHuman.org

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Sat Jun 02, 2012 1:19 pm

duststorm wrote:Prodecural techniques can also be very handy to avoid having to cache large amounts of data in memory or read it from disk.
Not entirely sure whether this can apply to terrain, as it would probably look weird if it is different every time you return to the same area. But for terrain textures and details like vegetation placement it could make a big difference if you could just randomly generate it instead of having to load it from data.
Every terrain instance will be generated by some specified arguments, such as a random number. It's cheap to save them, so it will stay unchanged when you come back.
It's the same with the vegetation placement. Maybe we can write a placement-algorithm, and save the parameters next to its terrain instance data.
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Mon Jun 04, 2012 4:01 am

Weekly Progress:
  • Almost finish extracting the LodManager from Terrain.
    Going to read height/delta once for all, as I explained in early discussion. This is a big step, or you can say it is a big revert. But I think it worth. After that the component will be stable enough and almost ready for alpha testing, maybe next weekend.
Issue:
Pls let me know if you have any opposite ideas about giving up reading LODs separately from disk. I'm not sure if it is necessary in some special occasion.
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

User avatar
Wolfmanfx
OGRE Team Member
OGRE Team Member
Posts: 1525
Joined: Fri Feb 03, 2006 10:37 pm
Location: Austria - Leoben
x 1
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by Wolfmanfx » Mon Jun 04, 2012 7:04 am

I think its important to stream in/out terrain data otherwise its not possible to render really huge terrains like gis data which could exceed your system ram.
0 x

User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by masterfalcon » Mon Jun 04, 2012 7:24 am

I agree. Streaming the terrain data is very important and shouldn't be abandoned. Think about mobiles and consoles for example where RAM is limited.

I'm not sure about keeping the entire set of delta data in memory. How large does it tend to be? Perhaps some of it could be loaded dynamically as well?
0 x

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Mon Jun 04, 2012 7:40 am

Wolfmanfx wrote:I think its important to stream in/out terrain data otherwise its not possible to render really huge terrains like gis data which could exceed your system ram.
My doubt is that, the current implementation still holds fully allocated HeightData/DeltaData, while just fill them time to time.
Should we also improve this, by allocating HeightData/DeltaData particially?
But in that way, all functions depending on the full HeightData will be failed, such as setHeight(x,y).
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Mon Jun 04, 2012 7:55 am

masterfalcon wrote:I agree. Streaming the terrain data is very important and shouldn't be abandoned. Think about mobiles and consoles for example where RAM is limited.

I'm not sure about keeping the entire set of delta data in memory. How large does it tend to be? Perhaps some of it could be loaded dynamically as well?
The delta data is as large as the height data.
The current implementation loads both of them dynamically. But it always holds the entire ram buffer. So I think it's no harm to read them in ram at once. Of course, still load dynamically into vertex buffer and into GPU.
As explained above, another solution is to allocate Height/Delta ram on demand, while it has some drawbacks.
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

User avatar
duststorm
Minaton
Posts: 921
Joined: Sat Jul 31, 2010 6:29 pm
Location: Belgium
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by duststorm » Mon Jun 04, 2012 12:30 pm

To make it really scaleable, you need to swap the terrain data in and out both on GPU RAM as on general RAM. Otherwise the terrain size will ultimately be limited to the amount of RAM in the machine.
With a real paging solution, the size of your terrain should only be limited by the amount of space on your hard disk.


A suggestion of a system you could use:

in RAM: Stream in neighbouring pages of current tile, so transition to any of these can happen smoothly. As soon as the camera enters another page, the neighbours that are not needed anymore are cleared, and new ones are streamed in.
The RAM part can probably be configured to specify how much neighbours are loaded exactly.

in GPU RAM: you probably only want to load in the details that are absolutely necessary to render. Meaning reduced LODs of far-off terrain, and the detail of the current page.

It's probably a good idea to change the notion of "current page" to two current pages, as there is always the possibility of standing on the border between two pages (or does this need to be expanded to 4 pages, as it is possible to stand in the point between four pages?) Maybe one larger page can be split into 4 smaller tiles so only the needed parts can be loaded.

Another configurable parameter can be drawing distance, which can influence the amount of pages or LODs loaded both in RAM and GPU. It's also possible to have a "distance detail" parameter to specify how much detail is reduced at specific distances.

Maybe the caching mechanism of the RAM can be modularized so custom caching strategies can be implemented instead of more general ones. A general one loads pages evenly in a circle around the current page, to account for the fact that the camera can move in any direction. Some applications might be able to optimize the caching because they know in advance which path the camera will follow (eg an animation sequence where the camera follows a spline, a racing game where you know where the track will lead, ...) and thus which pages to load next.


All these things might be too much to be in the scope of one GSoC project, but identifying possible extensions and uses for the paging terrain system can help you decide on an architecture to use.
0 x
Developer @ MakeHuman.org

PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG » Mon Jun 04, 2012 5:46 pm

xiaoxiangquan wrote:
Wolfmanfx wrote:I think its important to stream in/out terrain data otherwise its not possible to render really huge terrains like gis data which could exceed your system ram.
My doubt is that, the current implementation still holds fully allocated HeightData/DeltaData, while just fill them time to time.
Should we also improve this, by allocating HeightData/DeltaData particially?
But in that way, all functions depending on the full HeightData will be failed, such as setHeight(x,y).
I think it is neccesary to maintain setHeight(x,y) available as the terrain edition posibility is the personality of the current OgreTerrain. An approach could be to define a new flag (i.e. "editable") and Ogre::Terrain::setEditable(bool ed) for the terrain instance in order to enforce the full LODs loading when the application set the "edition mode" rather than a "simulation mode" or "playing mode". In "edition mode", some performance hiccups could be acceptable. In addition, for some mobiles or consoles with reduced memory it will always be possible to avoid the "edition mode".

Regards,
0 x
Paco, Spain

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Tue Jun 05, 2012 5:14 am

Thank duststorm and PacoRG! That makes sense.

I think I can stop the current work for a while, just make the original version (I mean, kuxv's work) works stably. The alpha testing will be ready this weekend.
The next two weeks I should deal with my final exams. During that I mainly writing some documents about the work.
And let's discuss more about the design clearly. Then we come to the stage 2, making improvements. :D
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Sun Jun 10, 2012 4:57 pm

Weekly Progress
I think I have almost finished the stage one:
xiaoxiangquan wrote:5.21~6.11 (3 weeks) Code review and bugfix, close most of the cases reported in the forum.
You can take it as an alpha release. Welcome to have a try :D

Known problem:
It will crack when you re-start the TerrainPaging Demo. It's an old bug, still not fixed. It must be some memory error.

Anyway, all the functions work quite well, both on windows and linux.

The next two weeks will be tough for me, due to the final exams. So I'll just write some documents about the work, and deal with the bugs finded.
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG » Tue Jun 12, 2012 6:05 pm

HI,

I have tested the last terrain revision in the "ogre_soc_tpi" repository (revision 825353ec5518). I compiled it in Visual C++ 2008.

I get the same crash when the example is started by second time from the SampleBrowser.exe.

I have modified the paging terrain example in order to create 5x5 flat pages (blankTerrain = true) and all seem to work well but the autoLOD system is stopped when the autoLOD flag is activated. In any way, once all the terrains are saved, the autoLOD mechanism starts and seems to work well.

After that, I uncommented "#define PAGING" in order to enable the paging system and start again the example. Now, when I move the camera not all the terrain pages are visible. The problem are showed in the screenshot:

Image

In adition, the example sometime crashed during the rendering operation. Therefore, some instabilities persist.

Regards,

Windows Vista 32, AMD athlon 64 X2 5200+, 4GB
NVIDIA GeForce GTX 560 TI
0 x
Paco, Spain

User avatar
duststorm
Minaton
Posts: 921
Joined: Sat Jul 31, 2010 6:29 pm
Location: Belgium
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by duststorm » Tue Jun 12, 2012 6:14 pm

I think these were some of the bugs that were appearing in kuxv's project and why it was judged not stable enough for integration in 1.8, is it not?
Those error reports seem familiar.
0 x
Developer @ MakeHuman.org

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Tue Jun 12, 2012 6:24 pm

Hi,
PacoRG wrote: I have modified the paging terrain example in order to create 5x5 flat pages (blankTerrain = true) and all seem to work well but the autoLOD system is stopped when the autoLOD flag is activated. In any way, once all the terrains are saved, the autoLOD mechanism starts and seems to work well.
The LodManager will not be initialed if the Terrain is not prepared from a file. Because all the data has been in memory and there is no need to read some level of LOD separately. If autoLOD is still wanted in this case, I can make it.
PacoRG wrote: After that, I uncommented "#define PAGING" in order to enable the paging system and start again the example. Now, when I move the camera not all the terrain pages are visible. The problem are showed in the screenshot:
Ops, the adaptation with the PAGING is still not finished.
PacoRG wrote: In adition, the example sometime crashed during the rendering operation. Therefore, some instabilities persist.
Is this occurs with PAGING opened? If not, could you give me more details?
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG » Tue Jun 12, 2012 6:29 pm

duststorm wrote:I think these were some of the bugs that were appearing in kuxv's project and why it was judged not stable enough for integration in 1.8, is it not?
Those error reports seem familiar.
The error is different. In the previous kuxv's project, the autoLOD system was incongruent according the camera position. Now, the autoLOD mechanism seem coherent with the camera position, however, some terrain pages are invisible.
0 x
Paco, Spain

PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG » Tue Jun 12, 2012 6:44 pm

xiaoxiangquan wrote:Hi,
PacoRG wrote: In adition, the example sometime crashed during the rendering operation. Therefore, some instabilities persist.
Is this occurs with PAGING opened? If not, could you give me more details?
I get this crash only when the paging mode is enabled. I did not know that the PAGING adaptation is still pending.

In any case, you are doing a good work, and the progress is clear. I hope my testing to be a help to you.

Thanks
0 x
Paco, Spain

xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan » Tue Jun 12, 2012 6:53 pm

PacoRG wrote:Hi,
I get this crash only when the paging mode is enabled. I did not know that the PAGING adaptation is still pending.
In any case, you are doing a good work, and the progress is clear. I hope my testing to be a help to you.
Thanks
[/quote]
Thank you!
I think PAGING will be ready for testing soon, too. :D
0 x
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon

User avatar
duststorm
Minaton
Posts: 921
Joined: Sat Jul 31, 2010 6:29 pm
Location: Belgium
Contact:

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by duststorm » Tue Jun 12, 2012 9:44 pm

Keep up the good work! :)
0 x
Developer @ MakeHuman.org

PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG » Tue Jun 12, 2012 11:01 pm

Hi,

I am afraid I have no good news.

After the 5x5 flat terrain pages have been saved I was flying some minutes over the terrain (PAGING disabled) and unexpectedly the demo crashed during the rendering operation:

Code: Select all

Unhandled exception at 0x5e093839 (RenderSystem_Direct3D9_d.dll) in SampleBrowser_d.exe: 0xC0000005: Access violation reading location 0x00000014


More details here

Regards,
0 x
Paco, Spain

Post Reply