Page 5 of 8

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jul 16, 2012 1:34 am
by xiaoxiangquan
Weekly Progress
Make the new terrain system work with paging ON
Fix the init problem, the sample can be restarted, now
Make AutoUpdateLodStrategy as singleton, and user can register his own strategy

Issue
With paging on, flicker occurs when loading a new page. I haven't found the main reason, yet.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Thu Jul 19, 2012 12:01 pm
by xiaoxiangquan
I have some thoughts about the second half of the program.
Though I have renamed the TerrainPaging sample to EndlessWorld and started the process of making the terrain generator as planed in my proposal, I think it's not good to slow down the key work, making the new terrain system stable enough to be merged into trunk.

It has been much stabler than before, but still far from perfect. For example,
  • The new format of terrain file is 3 times larger than before, as it is not well compressed
    The flicker when loading new terrain pages is still mysterious.
    Still lack of documentation
May I just put off the work of terrain generator and the new sample, and concentrate on pushing the process of release the key work? I promise the work delayed won't be discarded.
And, of course, welcome to testing the newest code. I haven't met previously known bugs myself for several days.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jul 21, 2012 4:47 pm
by PacoRG
Hi,

I just compiled the last revision.

I get again the CMake error: "Error in configuration process, project files may be invalid" (see post). The bug can be fixed by modifying the "ogre_soc_tpi/Samples/CMakeLists.txt" file replacing the line:

Code: Select all

set(SAMPLE_DEPENDENCIES ${SAMPLE_DEPENDENCIES} OgreEndlessWorld) 

with this other:

Code: Select all

set(SAMPLE_DEPENDENCIES ${SAMPLE_DEPENDENCIES} OgrePaging) 
I do not observe any problem except the flicker when a new terrain page becomes visible. Related to this I have observed that when the camera has no terrain under it the flicker seems a huge jump back of the camera, however, if the camera is over the terrain, I cannot be sure whether the camera jump but the flicker occurs.

IMHO I agree with you regarding your proposal of key work.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jul 21, 2012 5:04 pm
by xiaoxiangquan
PacoRG wrote:Hi,
I just compiled the last revision.
Thank you!
I incorrectly renamed the dependency setting, fixed now. So strange that no compile error occurs on my computer :(
I'll keep looking into the flicker problem. It seems a little troublesome.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sat Jul 21, 2012 8:51 pm
by PacoRG
xiaoxiangquan wrote: I incorrectly renamed the dependency setting, fixed now. So strange that no compile error occurs on my computer :(
Yes, I did not get compile errors in my computer neither, it only was a CMAKE error message during the configuration process.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jul 23, 2012 4:57 am
by xiaoxiangquan
Weekly Progress:
  • Extend StreamSerialiser to support Part-by-Part compression, apply it to terrain file, saves 65% disk space.
    Fix some bugs, such as camera-jump-back.
    PG-UP/DOWN to inc/dec LOD manually (revert, and enhance).
    Update wiki.
Issue:
  • Delay terrain generation work.
    Keep on looking into the flicker problem when loading new terrain pages

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jul 23, 2012 7:01 am
by masterfalcon
Great work! I've tried to reproduce the flicker but I can't see it. Though I have vsync enabled, perhaps it only happens at much higher frame rates?

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Mon Jul 23, 2012 4:29 pm
by PacoRG
masterfalcon wrote:I've tried to reproduce the flicker but I can't see it. Though I have vsync enabled, perhaps it only happens at much higher frame rates?
I have enabled the vSync and the flicker is observable too. It seems independent on the frame rate, the screen resolution and whether the fullscreen mode is enabled.

When I have tried to test the EndLess World example in OpenGL I get a crash when the example try to destroy an overlay not found.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jul 24, 2012 7:02 am
by xiaoxiangquan
According to my profiler, when a new terrain page loaded, some costly procedures will execute synchronously, such as:

Code: Select all

Terrain::load()
{
  ...
  //these take 0.15s
  checkLayers(true);
  createOrDestroyGPUColourMap();
  createOrDestroyGPUNormalMap();
  createOrDestroyGPULightmap();
  createOrDestroyGPUCompositeMap();
  ...
}
...
// the material is null when the page just loaded, so generation procedure will be called
Terrain::getMaterial()
{
  ...
  // these take more than 0.15s
  mMaterial = mMaterialGenerator->generate(this);
  ...
  mCompositeMapMaterial = mMaterialGenerator->generateForCompositeMap(this);
  ...
}
So a new terrain page will cause at least 0.3s (on my computer, debug mode). That's why the flicker occurs.
And if more pages are loaded (nearly) at once, the result will be accumulated.

I have tried to move some of the procedures to background, but it seems difficult, as they all do GPU operations.
Anyone has some ideas? Or just tolerate it?

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jul 24, 2012 7:28 am
by masterfalcon
Hmm, looks like a lot of GPU resources being created or modified during that time. Have you narrowed it down further within those functions? Is it the buffer write, texture allocation, texture blit?

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jul 24, 2012 7:31 am
by xiaoxiangquan
masterfalcon wrote:Hmm, looks like a lot of GPU resources being created or modified during that time. Have you narrowed it down further within those functions? Is it the buffer write, texture allocation, texture blit?
Not yet, do it right now.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jul 24, 2012 4:18 pm
by xiaoxiangquan
masterfalcon wrote:Hmm, looks like a lot of GPU resources being created or modified during that time. Have you narrowed it down further within those functions? Is it the buffer write, texture allocation, texture blit?

Code: Select all

Root::renderOneFrame ->
...
SceneManager::firePreFindVisibleObjects ->
...
Terrain::getMaterial ->
TerrainMaterialGeneratorA::addTechnique * 3 times ->
generateVertexProgram and generateFragmentProgram
...
Resource::load(bool background) * 6 times
Every load() takes 0.03s. I tried setting background as true, but no improvement.

Code: Select all

Root::renderOneFrame ->
...
Terrain::load ->
Terrain::createGPUBlendTextures and Terrain::createOrDestroyGPUCompositeMap ->
...
D3D9HardwarePixelBuffer::blitFromMemory * 2 times
Every blit() takes about 0.03s, too.

So loading a new terrain page cost 0.2s+ on these two operations.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Jul 24, 2012 6:26 pm
by xiaoxiangquan
PacoRG wrote: I have enabled the vSync and the flicker is observable too. It seems independent on the frame rate, the screen resolution and whether the fullscreen mode is enabled.
When I have tried to test the EndLess World example in OpenGL I get a crash when the example try to destroy an overlay not found.
Eh, I can not reproduce the crash. Both OpenGL and DX seem to work fine.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Wed Jul 25, 2012 6:29 am
by syedhs
Is the flicker just concern the display only (but FPS remains smooth) or is it actually jitter in frame rate that cause it to momentarily stop for a fraction of seconds..? The latter is the issue that Sinbad faces before, and I believe it was the main reason why he was not releasing Terrain+Paging for production uses..

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Wed Jul 25, 2012 6:39 am
by xiaoxiangquan
syedhs wrote:Is the flicker just concern the display only (but FPS remains smooth) or is it actually jitter in frame rate that cause it to momentarily stop for a fraction of seconds..? The latter is the issue that Sinbad faces before, and I believe it was the main reason why he was not releasing Terrain+Paging for production uses..
It's the latter. A single frame takes about 0.3s for a new terrain page to do initialization. The following frames will be smooth.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Wed Jul 25, 2012 3:55 pm
by Sadahar
Does a single page loading causes the jitter.. or when it loads 2-3?

Can you split the loading phase over frames? Lets say.. one load/blit operation and then put a mark to pass one frame.
(I know, can be tricky... but should be possible :D)

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Wed Jul 25, 2012 4:10 pm
by xiaoxiangquan
Sadahar wrote:Does a single page loading causes the jitter.. or when it loads 2-3?

Can you split the loading phase over frames? Lets say.. one load/blit operation and then put a mark to pass one frame.
(I know, can be tricky... but should be possible :D)
Yes, a single page does. To load 2 or 3 pages, the jitter effect will add up.
Splitting loading will be a little difficult, especially when several pages are loaded closely.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Wed Jul 25, 2012 5:44 pm
by PacoRG
xiaoxiangquan wrote:
PacoRG wrote: I have enabled the vSync and the flicker is observable too. It seems independent on the frame rate, the screen resolution and whether the fullscreen mode is enabled.
When I have tried to test the EndLess World example in OpenGL I get a crash when the example try to destroy an overlay not found.
Eh, I can not reproduce the crash. Both OpenGL and DX seem to work fine.
OK, I get the crash with all the examples in OpenGL. It is posible an incorrect installation in my system. I must revise it.

xiaoxiangquan wrote:
Sadahar wrote:Does a single page loading causes the jitter.. or when it loads 2-3?

Can you split the loading phase over frames? Lets say.. one load/blit operation and then put a mark to pass one frame.
(I know, can be tricky... but should be possible :D)
Yes, a single page does. To load 2 or 3 pages, the jitter effect will add up.
Splitting loading will be a little difficult, especially when several pages are loaded closely.
It will always be posible to separate the load of two or more pages by enqueing them and then applying a delay of 1 or 2 second between each load because the multiple load page coincidence will only occur in a disperse way along the time.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Wed Jul 25, 2012 5:52 pm
by xiaoxiangquan
PacoRG wrote: It will always be posible to separate the load of two or more pages by enqueing them and then applying a delay of 1 or 2 second between each load because the multiple load page coincidence will only occur in a disperse way along the time.
Thanks, Let me have a try to separate them.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Thu Jul 26, 2012 4:59 am
by syedhs
I hope you can really solve this jitter for good, because previous GSOC project had done lots of code factoring to solve it but apparently, it is still there.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Wed Aug 01, 2012 9:39 am
by xiaoxiangquan
I've committed two new revs to queue the page-loading, move material-generation to loading time and insert some time intervals to the process.
Now the page-loading seems a little smoother, especially when under release mode.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sun Aug 05, 2012 4:57 pm
by xiaoxiangquan
Weekly Progress:
Tried to make loading page smoother: queueing page-loadings, and divide material generation procedure.

To my knowledge, most of the loading time is spent on generating material. Looking the calling tree below:
Terrain::getMaterial
  • generate
    • addTechnique
      • generateVertexProgram
        generateFragmentProgram
      addTechnique
      • generateVertexProgram
        generateFragmentProgram
    generateForCompositeMap
    • addTechnique
      • generateVertexProgram
        generateFragmentProgram
Currently I divide generate and generateForCompositeMap into different workqueue request. On my computer either of them will cost less than 50ms, so the loading looks smooth enough.
I have also tried to divide all generateVertexPrograms and generateFragmentPrograms(either of them will be called 3 times in a generation process). Surely it becomes smoother, but the code complexity increases a lot and becomes difficult to read.
Please have a try if you have time, and tell me whether the current effect is acceptable.

Next week I'll startup the terrain generation work. I'm afraid the "Endless World" sample cannot be finished during the gsoc program, but surely I'll keep on and make it complete.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Sun Aug 05, 2012 5:09 pm
by xiaoxiangquan
See How_to_Testing in wiki.

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Aug 07, 2012 6:16 am
by masterfalcon
I need to test it out here but that sounds like a good plan. Keeping the code readable will be very important for maintenance. How much of the Endless World sample do you think you will be able to get finished?

Re: [GSoC 2012 - Accepted] Improve and Demo the Terrain Syst

Posted: Tue Aug 07, 2012 6:45 am
by xiaoxiangquan
masterfalcon wrote:I need to test it out here but that sounds like a good plan. Keeping the code readable will be very important for maintenance. How much of the Endless World sample do you think you will be able to get finished?
I hope I can finish the alpha version in two weeks, just before the official Pencil-Down. And then make it better in another two weeks.