Page 3 of 8

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Thu May 31, 2012 5:27 am
by masterfalcon
My only concern is the possibility of performance hiccups when the LOD changes.

Perhaps we could attempt to predict when LOD will change and load on the fly?

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Thu May 31, 2012 5:24 pm
by duststorm
Are there no such strategies already implemented in other engines, that predict the next pages to load and stream them in before they are actually necessary?
If you are able to allocate enough RAM space to load all neighbouring pages in, the camera can move to any neighbour, then the only assumption you make is that the camera will not move faster to another page than that pages can be loaded from disk. I believe this is the sort of assumption that is made in some large outdoor rendering setups. (Just Cause 2 comes to mind: the only loading you ever see there is if you teleport from one area to the other. When traveling through the world regularly you never notice any loading, as the necessary data is streamed in memory just before it is needed.)

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jun 02, 2012 3:22 am
by xiaoxiangquan
masterfalcon wrote:My only concern is the possibility of performance hiccups when the LOD changes.
Perhaps we could attempt to predict when LOD will change and load on the fly?
Ehh, the topic reply notification seems broken.

Performance hiccups is indeed one of the problems, which you can notice by flying around fast. So I want to reduce the loading-from-disk step when the LOD changes. As I mentioned above, this step doesn't save any memories, as well as performance.

We can predict that one LOD seems to be needed by watching the DeltaData. See:
...
|
| <-- The biggest delta data is noticeable!
| <-- The LOD seems to be needed?
|
...
For this reason, the whole copy of delta data should be always available, too.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jun 02, 2012 8:57 am
by aguru
Quite unrelated to the current discussion, but I wondered if you are looking for any beautiful heightmap data for the showcase demo? We have some experience with those.
http://www.ogre3d.org/forums/viewtopic. ... 25#p368126

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jun 02, 2012 9:44 am
by xiaoxiangquan
aguru wrote:Quite unrelated to the current discussion, but I wondered if you are looking for any beautiful heightmap data for the showcase demo? We have some experience with those.
http://www.ogre3d.org/forums/viewtopic. ... 25#p368126
Thank you so much! The island is very beautiful!
In my showcase demo, heightmap will be generated randomly. I hope it look good :D

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jun 02, 2012 10:21 am
by aguru
Yes, generating it randomly is a good idea, that way distribution size won't get too large ;)

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jun 02, 2012 12:59 pm
by duststorm
Prodecural techniques can also be very handy to avoid having to cache large amounts of data in memory or read it from disk.
Not entirely sure whether this can apply to terrain, as it would probably look weird if it is different every time you return to the same area. But for terrain textures and details like vegetation placement it could make a big difference if you could just randomly generate it instead of having to load it from data.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jun 02, 2012 1:19 pm
by xiaoxiangquan
duststorm wrote:Prodecural techniques can also be very handy to avoid having to cache large amounts of data in memory or read it from disk.
Not entirely sure whether this can apply to terrain, as it would probably look weird if it is different every time you return to the same area. But for terrain textures and details like vegetation placement it could make a big difference if you could just randomly generate it instead of having to load it from data.
Every terrain instance will be generated by some specified arguments, such as a random number. It's cheap to save them, so it will stay unchanged when you come back.
It's the same with the vegetation placement. Maybe we can write a placement-algorithm, and save the parameters next to its terrain instance data.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jun 04, 2012 4:01 am
by xiaoxiangquan
Weekly Progress:
  • Almost finish extracting the LodManager from Terrain.
    Going to read height/delta once for all, as I explained in early discussion. This is a big step, or you can say it is a big revert. But I think it worth. After that the component will be stable enough and almost ready for alpha testing, maybe next weekend.
Issue:
Pls let me know if you have any opposite ideas about giving up reading LODs separately from disk. I'm not sure if it is necessary in some special occasion.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jun 04, 2012 7:04 am
by Wolfmanfx
I think its important to stream in/out terrain data otherwise its not possible to render really huge terrains like gis data which could exceed your system ram.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jun 04, 2012 7:24 am
by masterfalcon
I agree. Streaming the terrain data is very important and shouldn't be abandoned. Think about mobiles and consoles for example where RAM is limited.

I'm not sure about keeping the entire set of delta data in memory. How large does it tend to be? Perhaps some of it could be loaded dynamically as well?

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jun 04, 2012 7:40 am
by xiaoxiangquan
Wolfmanfx wrote:I think its important to stream in/out terrain data otherwise its not possible to render really huge terrains like gis data which could exceed your system ram.
My doubt is that, the current implementation still holds fully allocated HeightData/DeltaData, while just fill them time to time.
Should we also improve this, by allocating HeightData/DeltaData particially?
But in that way, all functions depending on the full HeightData will be failed, such as setHeight(x,y).

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jun 04, 2012 7:55 am
by xiaoxiangquan
masterfalcon wrote:I agree. Streaming the terrain data is very important and shouldn't be abandoned. Think about mobiles and consoles for example where RAM is limited.

I'm not sure about keeping the entire set of delta data in memory. How large does it tend to be? Perhaps some of it could be loaded dynamically as well?
The delta data is as large as the height data.
The current implementation loads both of them dynamically. But it always holds the entire ram buffer. So I think it's no harm to read them in ram at once. Of course, still load dynamically into vertex buffer and into GPU.
As explained above, another solution is to allocate Height/Delta ram on demand, while it has some drawbacks.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jun 04, 2012 12:30 pm
by duststorm
To make it really scaleable, you need to swap the terrain data in and out both on GPU RAM as on general RAM. Otherwise the terrain size will ultimately be limited to the amount of RAM in the machine.
With a real paging solution, the size of your terrain should only be limited by the amount of space on your hard disk.


A suggestion of a system you could use:

in RAM: Stream in neighbouring pages of current tile, so transition to any of these can happen smoothly. As soon as the camera enters another page, the neighbours that are not needed anymore are cleared, and new ones are streamed in.
The RAM part can probably be configured to specify how much neighbours are loaded exactly.

in GPU RAM: you probably only want to load in the details that are absolutely necessary to render. Meaning reduced LODs of far-off terrain, and the detail of the current page.

It's probably a good idea to change the notion of "current page" to two current pages, as there is always the possibility of standing on the border between two pages (or does this need to be expanded to 4 pages, as it is possible to stand in the point between four pages?) Maybe one larger page can be split into 4 smaller tiles so only the needed parts can be loaded.

Another configurable parameter can be drawing distance, which can influence the amount of pages or LODs loaded both in RAM and GPU. It's also possible to have a "distance detail" parameter to specify how much detail is reduced at specific distances.

Maybe the caching mechanism of the RAM can be modularized so custom caching strategies can be implemented instead of more general ones. A general one loads pages evenly in a circle around the current page, to account for the fact that the camera can move in any direction. Some applications might be able to optimize the caching because they know in advance which path the camera will follow (eg an animation sequence where the camera follows a spline, a racing game where you know where the track will lead, ...) and thus which pages to load next.


All these things might be too much to be in the scope of one GSoC project, but identifying possible extensions and uses for the paging terrain system can help you decide on an architecture to use.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jun 04, 2012 5:46 pm
by PacoRG
xiaoxiangquan wrote:
Wolfmanfx wrote:I think its important to stream in/out terrain data otherwise its not possible to render really huge terrains like gis data which could exceed your system ram.
My doubt is that, the current implementation still holds fully allocated HeightData/DeltaData, while just fill them time to time.
Should we also improve this, by allocating HeightData/DeltaData particially?
But in that way, all functions depending on the full HeightData will be failed, such as setHeight(x,y).
I think it is neccesary to maintain setHeight(x,y) available as the terrain edition posibility is the personality of the current OgreTerrain. An approach could be to define a new flag (i.e. "editable") and Ogre::Terrain::setEditable(bool ed) for the terrain instance in order to enforce the full LODs loading when the application set the "edition mode" rather than a "simulation mode" or "playing mode". In "edition mode", some performance hiccups could be acceptable. In addition, for some mobiles or consoles with reduced memory it will always be possible to avoid the "edition mode".

Regards,

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 05, 2012 5:14 am
by xiaoxiangquan
Thank duststorm and PacoRG! That makes sense.

I think I can stop the current work for a while, just make the original version (I mean, kuxv's work) works stably. The alpha testing will be ready this weekend.
The next two weeks I should deal with my final exams. During that I mainly writing some documents about the work.
And let's discuss more about the design clearly. Then we come to the stage 2, making improvements. :D

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sun Jun 10, 2012 4:57 pm
by xiaoxiangquan
Weekly Progress
I think I have almost finished the stage one:
xiaoxiangquan wrote:5.21~6.11 (3 weeks) Code review and bugfix, close most of the cases reported in the forum.
You can take it as an alpha release. Welcome to have a try :D

Known problem:
It will crack when you re-start the TerrainPaging Demo. It's an old bug, still not fixed. It must be some memory error.

Anyway, all the functions work quite well, both on windows and linux.

The next two weeks will be tough for me, due to the final exams. So I'll just write some documents about the work, and deal with the bugs finded.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 6:05 pm
by PacoRG
HI,

I have tested the last terrain revision in the "ogre_soc_tpi" repository (revision 825353ec5518). I compiled it in Visual C++ 2008.

I get the same crash when the example is started by second time from the SampleBrowser.exe.

I have modified the paging terrain example in order to create 5x5 flat pages (blankTerrain = true) and all seem to work well but the autoLOD system is stopped when the autoLOD flag is activated. In any way, once all the terrains are saved, the autoLOD mechanism starts and seems to work well.

After that, I uncommented "#define PAGING" in order to enable the paging system and start again the example. Now, when I move the camera not all the terrain pages are visible. The problem are showed in the screenshot:

Image

In adition, the example sometime crashed during the rendering operation. Therefore, some instabilities persist.

Regards,

Windows Vista 32, AMD athlon 64 X2 5200+, 4GB
NVIDIA GeForce GTX 560 TI

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 6:14 pm
by duststorm
I think these were some of the bugs that were appearing in kuxv's project and why it was judged not stable enough for integration in 1.8, is it not?
Those error reports seem familiar.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 6:24 pm
by xiaoxiangquan
Hi,
PacoRG wrote: I have modified the paging terrain example in order to create 5x5 flat pages (blankTerrain = true) and all seem to work well but the autoLOD system is stopped when the autoLOD flag is activated. In any way, once all the terrains are saved, the autoLOD mechanism starts and seems to work well.
The LodManager will not be initialed if the Terrain is not prepared from a file. Because all the data has been in memory and there is no need to read some level of LOD separately. If autoLOD is still wanted in this case, I can make it.
PacoRG wrote: After that, I uncommented "#define PAGING" in order to enable the paging system and start again the example. Now, when I move the camera not all the terrain pages are visible. The problem are showed in the screenshot:
Ops, the adaptation with the PAGING is still not finished.
PacoRG wrote: In adition, the example sometime crashed during the rendering operation. Therefore, some instabilities persist.
Is this occurs with PAGING opened? If not, could you give me more details?

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 6:29 pm
by PacoRG
duststorm wrote:I think these were some of the bugs that were appearing in kuxv's project and why it was judged not stable enough for integration in 1.8, is it not?
Those error reports seem familiar.
The error is different. In the previous kuxv's project, the autoLOD system was incongruent according the camera position. Now, the autoLOD mechanism seem coherent with the camera position, however, some terrain pages are invisible.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 6:44 pm
by PacoRG
xiaoxiangquan wrote:Hi,
PacoRG wrote: In adition, the example sometime crashed during the rendering operation. Therefore, some instabilities persist.
Is this occurs with PAGING opened? If not, could you give me more details?
I get this crash only when the paging mode is enabled. I did not know that the PAGING adaptation is still pending.

In any case, you are doing a good work, and the progress is clear. I hope my testing to be a help to you.

Thanks

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 6:53 pm
by xiaoxiangquan
PacoRG wrote:Hi,
I get this crash only when the paging mode is enabled. I did not know that the PAGING adaptation is still pending.
In any case, you are doing a good work, and the progress is clear. I hope my testing to be a help to you.
Thanks
[/quote]
Thank you!
I think PAGING will be ready for testing soon, too. :D

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 9:44 pm
by duststorm
Keep up the good work! :)

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jun 12, 2012 11:01 pm
by PacoRG
Hi,

I am afraid I have no good news.

After the 5x5 flat terrain pages have been saved I was flying some minutes over the terrain (PAGING disabled) and unexpectedly the demo crashed during the rendering operation:

Code: Select all

Unhandled exception at 0x5e093839 (RenderSystem_Direct3D9_d.dll) in SampleBrowser_d.exe: 0xC0000005: Access violation reading location 0x00000014


More details here

Regards,