[GSoC 2012 - Accepted] Improve and Demo the Terrain System

Threads related to Google Summer of Code
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

Weekly Progress
Make the new terrain system work with paging ON
Fix the init problem, the sample can be restarted, now
Make AutoUpdateLodStrategy as singleton, and user can register his own strategy

Issue
With paging on, flicker occurs when loading a new page. I haven't found the main reason, yet.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

I have some thoughts about the second half of the program.
Though I have renamed the TerrainPaging sample to EndlessWorld and started the process of making the terrain generator as planed in my proposal, I think it's not good to slow down the key work, making the new terrain system stable enough to be merged into trunk.

It has been much stabler than before, but still far from perfect. For example,
  • The new format of terrain file is 3 times larger than before, as it is not well compressed
    The flicker when loading new terrain pages is still mysterious.
    Still lack of documentation
May I just put off the work of terrain generator and the new sample, and concentrate on pushing the process of release the key work? I promise the work delayed won't be discarded.
And, of course, welcome to testing the newest code. I haven't met previously known bugs myself for several days.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain
x 1

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG »

Hi,

I just compiled the last revision.

I get again the CMake error: "Error in configuration process, project files may be invalid" (see post). The bug can be fixed by modifying the "ogre_soc_tpi/Samples/CMakeLists.txt" file replacing the line:

Code: Select all

set(SAMPLE_DEPENDENCIES ${SAMPLE_DEPENDENCIES} OgreEndlessWorld) 

with this other:

Code: Select all

set(SAMPLE_DEPENDENCIES ${SAMPLE_DEPENDENCIES} OgrePaging) 
I do not observe any problem except the flicker when a new terrain page becomes visible. Related to this I have observed that when the camera has no terrain under it the flicker seems a huge jump back of the camera, however, if the camera is over the terrain, I cannot be sure whether the camera jump but the flicker occurs.

IMHO I agree with you regarding your proposal of key work.
Paco, Spain
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

PacoRG wrote:Hi,
I just compiled the last revision.
Thank you!
I incorrectly renamed the dependency setting, fixed now. So strange that no compile error occurs on my computer :(
I'll keep looking into the flicker problem. It seems a little troublesome.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain
x 1

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG »

xiaoxiangquan wrote: I incorrectly renamed the dependency setting, fixed now. So strange that no compile error occurs on my computer :(
Yes, I did not get compile errors in my computer neither, it only was a CMAKE error message during the configuration process.
Paco, Spain
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

Weekly Progress:
  • Extend StreamSerialiser to support Part-by-Part compression, apply it to terrain file, saves 65% disk space.
    Fix some bugs, such as camera-jump-back.
    PG-UP/DOWN to inc/dec LOD manually (revert, and enhance).
    Update wiki.
Issue:
  • Delay terrain generation work.
    Keep on looking into the flicker problem when loading new terrain pages
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
x 126

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by masterfalcon »

Great work! I've tried to reproduce the flicker but I can't see it. Though I have vsync enabled, perhaps it only happens at much higher frame rates?
PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain
x 1

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG »

masterfalcon wrote:I've tried to reproduce the flicker but I can't see it. Though I have vsync enabled, perhaps it only happens at much higher frame rates?
I have enabled the vSync and the flicker is observable too. It seems independent on the frame rate, the screen resolution and whether the fullscreen mode is enabled.

When I have tried to test the EndLess World example in OpenGL I get a crash when the example try to destroy an overlay not found.
Paco, Spain
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

According to my profiler, when a new terrain page loaded, some costly procedures will execute synchronously, such as:

Code: Select all

Terrain::load()
{
  ...
  //these take 0.15s
  checkLayers(true);
  createOrDestroyGPUColourMap();
  createOrDestroyGPUNormalMap();
  createOrDestroyGPULightmap();
  createOrDestroyGPUCompositeMap();
  ...
}
...
// the material is null when the page just loaded, so generation procedure will be called
Terrain::getMaterial()
{
  ...
  // these take more than 0.15s
  mMaterial = mMaterialGenerator->generate(this);
  ...
  mCompositeMapMaterial = mMaterialGenerator->generateForCompositeMap(this);
  ...
}
So a new terrain page will cause at least 0.3s (on my computer, debug mode). That's why the flicker occurs.
And if more pages are loaded (nearly) at once, the result will be accumulated.

I have tried to move some of the procedures to background, but it seems difficult, as they all do GPU operations.
Anyone has some ideas? Or just tolerate it?
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
x 126

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by masterfalcon »

Hmm, looks like a lot of GPU resources being created or modified during that time. Have you narrowed it down further within those functions? Is it the buffer write, texture allocation, texture blit?
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

masterfalcon wrote:Hmm, looks like a lot of GPU resources being created or modified during that time. Have you narrowed it down further within those functions? Is it the buffer write, texture allocation, texture blit?
Not yet, do it right now.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

masterfalcon wrote:Hmm, looks like a lot of GPU resources being created or modified during that time. Have you narrowed it down further within those functions? Is it the buffer write, texture allocation, texture blit?

Code: Select all

Root::renderOneFrame ->
...
SceneManager::firePreFindVisibleObjects ->
...
Terrain::getMaterial ->
TerrainMaterialGeneratorA::addTechnique * 3 times ->
generateVertexProgram and generateFragmentProgram
...
Resource::load(bool background) * 6 times
Every load() takes 0.03s. I tried setting background as true, but no improvement.

Code: Select all

Root::renderOneFrame ->
...
Terrain::load ->
Terrain::createGPUBlendTextures and Terrain::createOrDestroyGPUCompositeMap ->
...
D3D9HardwarePixelBuffer::blitFromMemory * 2 times
Every blit() takes about 0.03s, too.

So loading a new terrain page cost 0.2s+ on these two operations.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

PacoRG wrote: I have enabled the vSync and the flicker is observable too. It seems independent on the frame rate, the screen resolution and whether the fullscreen mode is enabled.
When I have tried to test the EndLess World example in OpenGL I get a crash when the example try to destroy an overlay not found.
Eh, I can not reproduce the crash. Both OpenGL and DX seem to work fine.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
User avatar
syedhs
Silver Sponsor
Silver Sponsor
Posts: 2703
Joined: Mon Aug 29, 2005 3:24 pm
Location: Kuala Lumpur, Malaysia
x 51

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by syedhs »

Is the flicker just concern the display only (but FPS remains smooth) or is it actually jitter in frame rate that cause it to momentarily stop for a fraction of seconds..? The latter is the issue that Sinbad faces before, and I believe it was the main reason why he was not releasing Terrain+Paging for production uses..
A willow deeply scarred, somebody's broken heart
And a washed-out dream
They follow the pattern of the wind, ya' see
Cause they got no place to be
That's why I'm starting with me
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

syedhs wrote:Is the flicker just concern the display only (but FPS remains smooth) or is it actually jitter in frame rate that cause it to momentarily stop for a fraction of seconds..? The latter is the issue that Sinbad faces before, and I believe it was the main reason why he was not releasing Terrain+Paging for production uses..
It's the latter. A single frame takes about 0.3s for a new terrain page to do initialization. The following frames will be smooth.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
Sadahar
Gnoblar
Posts: 3
Joined: Sun Jul 22, 2012 6:30 pm

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by Sadahar »

Does a single page loading causes the jitter.. or when it loads 2-3?

Can you split the loading phase over frames? Lets say.. one load/blit operation and then put a mark to pass one frame.
(I know, can be tricky... but should be possible :D)
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

Sadahar wrote:Does a single page loading causes the jitter.. or when it loads 2-3?

Can you split the loading phase over frames? Lets say.. one load/blit operation and then put a mark to pass one frame.
(I know, can be tricky... but should be possible :D)
Yes, a single page does. To load 2 or 3 pages, the jitter effect will add up.
Splitting loading will be a little difficult, especially when several pages are loaded closely.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
PacoRG
Kobold
Posts: 26
Joined: Fri May 23, 2008 10:46 pm
Location: Spain
x 1

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by PacoRG »

xiaoxiangquan wrote:
PacoRG wrote: I have enabled the vSync and the flicker is observable too. It seems independent on the frame rate, the screen resolution and whether the fullscreen mode is enabled.
When I have tried to test the EndLess World example in OpenGL I get a crash when the example try to destroy an overlay not found.
Eh, I can not reproduce the crash. Both OpenGL and DX seem to work fine.
OK, I get the crash with all the examples in OpenGL. It is posible an incorrect installation in my system. I must revise it.

xiaoxiangquan wrote:
Sadahar wrote:Does a single page loading causes the jitter.. or when it loads 2-3?

Can you split the loading phase over frames? Lets say.. one load/blit operation and then put a mark to pass one frame.
(I know, can be tricky... but should be possible :D)
Yes, a single page does. To load 2 or 3 pages, the jitter effect will add up.
Splitting loading will be a little difficult, especially when several pages are loaded closely.
It will always be posible to separate the load of two or more pages by enqueing them and then applying a delay of 1 or 2 second between each load because the multiple load page coincidence will only occur in a disperse way along the time.
Paco, Spain
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

PacoRG wrote: It will always be posible to separate the load of two or more pages by enqueing them and then applying a delay of 1 or 2 second between each load because the multiple load page coincidence will only occur in a disperse way along the time.
Thanks, Let me have a try to separate them.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
User avatar
syedhs
Silver Sponsor
Silver Sponsor
Posts: 2703
Joined: Mon Aug 29, 2005 3:24 pm
Location: Kuala Lumpur, Malaysia
x 51

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by syedhs »

I hope you can really solve this jitter for good, because previous GSOC project had done lots of code factoring to solve it but apparently, it is still there.
A willow deeply scarred, somebody's broken heart
And a washed-out dream
They follow the pattern of the wind, ya' see
Cause they got no place to be
That's why I'm starting with me
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

I've committed two new revs to queue the page-loading, move material-generation to loading time and insert some time intervals to the process.
Now the page-loading seems a little smoother, especially when under release mode.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

Weekly Progress:
Tried to make loading page smoother: queueing page-loadings, and divide material generation procedure.

To my knowledge, most of the loading time is spent on generating material. Looking the calling tree below:
Terrain::getMaterial
  • generate
    • addTechnique
      • generateVertexProgram
        generateFragmentProgram
      addTechnique
      • generateVertexProgram
        generateFragmentProgram
    generateForCompositeMap
    • addTechnique
      • generateVertexProgram
        generateFragmentProgram
Currently I divide generate and generateForCompositeMap into different workqueue request. On my computer either of them will cost less than 50ms, so the loading looks smooth enough.
I have also tried to divide all generateVertexPrograms and generateFragmentPrograms(either of them will be called 3 times in a generation process). Surely it becomes smoother, but the code complexity increases a lot and becomes difficult to read.
Please have a try if you have time, and tell me whether the current effect is acceptable.

Next week I'll startup the terrain generation work. I'm afraid the "Endless World" sample cannot be finished during the gsoc program, but surely I'll keep on and make it complete.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

See How_to_Testing in wiki.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon
User avatar
masterfalcon
OGRE Team Member
OGRE Team Member
Posts: 4270
Joined: Sun Feb 25, 2007 4:56 am
Location: Bloomington, MN
x 126

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by masterfalcon »

I need to test it out here but that sounds like a good plan. Keeping the code readable will be very important for maintenance. How much of the Endless World sample do you think you will be able to get finished?
xiaoxiangquan
Google Summer of Code Student
Google Summer of Code Student
Posts: 102
Joined: Tue Mar 20, 2012 3:20 am
Location: China
x 13

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Post by xiaoxiangquan »

masterfalcon wrote:I need to test it out here but that sounds like a good plan. Keeping the code readable will be very important for maintenance. How much of the Endless World sample do you think you will be able to get finished?
I hope I can finish the alpha version in two weeks, just before the official Pencil-Down. And then make it better in another two weeks.
Google Summer of Code 2012 Student
Topic: "Improve and Demo the Terrain System"
Project links: Project thread, WIKI page, Code fork for the project
Mentor: masterfalcon