OGRE clipmaps terrain sample

Post by **Assaf Raman** » Sat Oct 14, 2006 4:51 am

I am interested in clipmaps terrain algorithm for about a year.

I didn’t find the time to implement it in OGRE until now.

As I wrote in this thread:
http://www.ogre3d.org/phpBB2addons/viewtopic.php?t=2425

3 day ago I found some code that implements a basic clipmaps terrain. (A link to the original code is there in the thread).

I spent the last 2 day converting it to OGRE.

Here are some screenshots:

Here is the code + exe:
http://www.upload2.net/page/download/nt ... n.zip.html

Also – I think I found a bug with OGRE direct 3D implementation when using “addElement”:
If you use vertex offset – it doesn’t work.
But if you don’t use it – the openGL FPS goes way down…

For now I am checking which render system it is and acting accordingly.

Well, I don’t know when I will have more time for this code – so I decided publish it “half baked” than not publish it.

If someone is willing to clean and fix it – go ahead and post the fixed code here in this thread.

[EDIT]

No bug in OGRE after all. Scroll down this thread for the fixed code.
[/EDIT]

[EDIT]
Here is the EXE with basic OGRE dlls – so you can just download, unzip and run:
http://www.upload2.net/page/download/n1 ... e.zip.html
[/EDIT]

Post by **Falagard** » Sat Oct 14, 2006 5:09 am

Sweet!

Alexander · Post by **Alexander** » Sat Oct 14, 2006 6:47 am

Very exciting assaf!

I'll have to check this out immediately. Does this calculate on the GPU or is it just a port of the original code sample?

Post by **Assaf Raman** » Sat Oct 14, 2006 12:14 pm

Just a port of the original code sample - for now.

Post by **sinbad** » Sat Oct 14, 2006 4:34 pm

Also – I think I found a bug with OGRE direct 3D implementation when using “addElement”:
If you use vertex offset – it doesn’t work.
But if you don’t use it – the openGL FPS goes way down…

Not sure what you mean, we use offsets all the time all over the OGRE code with addElement. Can you give a specific example? Perhaps you just broke the D3D9 rules and left a gap in the vertex declaration?

Post by **Assaf Raman** » Sat Oct 14, 2006 4:46 pm

If you want to see the offset problem -
1. download the code.
2. compile and see the result on both DirectX and OpenGL.
3. open the file: "ogrenew\Samples\Clipmap\include\clipmap.h"
4. change line 91 in the file to be:

Code: Select all

bool directXRenderSystem = "Direct3D9 Rendering Subsystem" != renderSystemName;

you will see the problem - in directX it will stop working at all and in OpenGL the FPS will be 20% from before.

Post by **sinbad** » Sat Oct 14, 2006 5:19 pm

Had a quick look - I think you're using the offset for the wrong purpose.

The offset is there to define the start within a single vertex. DirectX will not let you leave a big empty gap at the start of a vertex declaration.

To start rendering part way through a vertex buffer, use the vertexStart attribute of VertexData instead.

Post by **Assaf Raman** » Sat Oct 14, 2006 5:47 pm

You are right. Thanks for the help.

here is the update code + exe that does not check for render system:
http://www.upload2.net/page/download/8p ... 2.zip.html

jsommers · Post by **jsommers** » Sun Oct 15, 2006 8:27 am

Any further plans on taking this technology. Maybe a full blown scene manager?

Post by **Assaf Raman** » Sun Oct 15, 2006 7:16 pm

My next stage is to try the GPU gems 2 article extension – I will try to use a shader to draw the terrain.

I found some code here:
http://www.welter-4d.de/CG-competitionWS0405/index.html

Here are some screenshots:

I converted the code to windows (about 2 hours work) – here is the EXE – just download and run:
http://www.upload2.net/page/download/13 ... n.zip.html

Here are some screenshots of my windows version:

The code is about the same as the original, but if you need the

solution - here you go:
http://www.upload2.net/page/download/4P ... 5.zip.html

My next step will be to try to convert it to OGRE.

Anyone wants to help?

Alexander · Post by **Alexander** » Sun Oct 15, 2006 9:46 pm

Good luck with this Assaf.

In my team's and my opinion, geometry clipmaps are currently the most promising terrain rendering technology and it is very exciting seeing this come to Ogre (we had planned on having to add it ourself down the line).

While I do not have the resources to lend to the effort at the moment I wish you good luck with this and will follow closely.

Post by **tuan kuranes** » Mon Oct 16, 2006 10:07 am

- Be aware that geometry clipmap is patented :
http://earthbrowser.blogspot.com/2006/0 ... ipmap.html

- Unless using Vertex shader 3 (vertex texture fetch availaible only on nvidia) or R2VB from ati (not)that's some nearly continous update, using a llot of cpu-GPU bandwith (unless using a high threshold, but then paging would be as good.). Maybe next opengl and dx10 let's us have a standard way to render into vertex buffers...

- Here's some improvement on original clipmap : http://www.robertwrose.com/cg/terrain/
http://www.robertwrose.com/2005/06/geom ... pmaps.html

- Isn't the ogre planet scene manager posted once here use clipmap... search forums for source links.

From my point of view the most interesting part is the pyramidal - interpolation compression scheme (which is not patented btw), that gives very good results on heightmap. You didn't work yet on that or do you plan to do so ?
I'll intend to add that to plsm3.

Alexander · Post by **Alexander** » Mon Oct 16, 2006 10:41 am

The patent issue has been brought up before (on gamedev I believe).
While it's likely a defensive measure you could choose to implement all but one of the claims of the patent to avoid infringement.
You could also contact Microsoft yourself and ask permission.

If I were to do the implementation I would likey save the CPU-centric version of the system as a fallback to the SM3.

I've spoken to Robert Rose before regarding his implementation and he was pretty forthcoming. Perhaps he could offer some source code from his "not-technically-geometry clipmaps".

Post by **Assaf Raman** » Mon Oct 16, 2006 10:58 am

I know that the clipmap is patented. I don't think that it is a problem – if I will use it – I will change it – so the patent will not be affective.

Regarding dx10 – I want to use a geometric shader to create the geometry completely on the GPU – so no cpu-GPU bandwith for geometry. And Tuan is right about vertex texture fetch.

I like this improvement of the original clipmap:
http://www.zib.de/clasen/?page_id=4

"Pyramidal - interpolation compression scheme" sounds great – but I didn't understand the article about it, and I am not sure it will be as good for the texture of the terrain. Also – I want the textures natively compressed on the GPU – to have the data on the GPU in DDS format. What I am going to do is to create a directory structure like this – the depth of the dir will be the LOD, I will have DDS files for each LOD for the texture and PNG files for terrain heights, I don't care about disk space. I will have a thread that will load this data. I will have a cache that will be used to save loads. If data is not available – I will use the parent LOD. This is the general idea. This way I will also be able to have different LOD levels for different parts of the terrain.

Regarding the planetary rendering scene manager – well I think it is dead – have a look here:
http://www.ogre3d.org/phpBB2/viewtopic.php?t=10773
If someone has the code – or a thread with the code – please post it here…

I will be also happy to get code from Robert Rose if he will give it.

Post by **tuan kuranes** » Mon Oct 16, 2006 11:15 am

I know that the clipmap is patented. I don't think that it is a problem

Depends on you lawyer section in your company... have to be known and specified in any readme.txt you provide with your code IMHO.

"Pyramidal - interpolation compression scheme"

That's for heightmap data, not textures. And it achieve really, really impressive compression rates.
Puget sound from 537MB to 8.5MB...

I don't care about disk space

Then care about CPU and GPU memory use...

Regarding the planetary rendering scene manager – well I think it is dead – have a look here:
http://www.ogre3d.org/phpBB2/viewtopic.php?t=10773
If someone has the code – or a thread with the code – please post it here…

Lucky you are : I have a version that I may have modified (moslty to make it Dagon compatible)... but cannot find original zip or thing... so here it is (source and media only):
clipmap scene manager
clipmap demo
clipmap converter
clipmap media

curantil · Post by **curantil** » Mon Oct 16, 2006 11:53 am

Assaf Raman wrote: Regarding the planetary rendering scene manager – well I think it is dead – have a look here:
http://www.ogre3d.org/phpBB2/viewtopic.php?t=10773
If someone has the code – or a thread with the code – please post it here…

I will be also happy to get code from Robert Rose if he will give it.

(I'm the author of that planetary scenemanager)
For the moment I don't work on the planetary scenemanager anymore. I know I promised to make it better, but I don't have the time for it anymore. There is a lot room for improvement and userfriendlyness.

The project is offline because I finished my studies and the webspace was of my university. But I'll look for some webspace to put it on so you can get it. If someone knows a good free provider please PM me.

Post by **tuan kuranes** » Mon Oct 16, 2006 12:09 pm

@curantil: http://pages.google.com ?
With very little work (readme, installation, license and todo) it might be ready to be turned into an OgreAddon IMHO.

Post by **Assaf Raman** » Mon Oct 16, 2006 1:14 pm

Tuan - thanks for the code. I guessed you had it.

Curantil - what was the status when you stoped working on the planetary scenemanager project?
If you want to share files in the forum - use - http://www.upload2.net/

I don't want to "reinvent the wheel"… So if the planetary scenemanager project is a good starting point - I will use it.
Curantil - just to make clear - what license is the code under?

curantil · Post by **curantil** » Mon Oct 16, 2006 10:45 pm

Assaf - When I stopped working I had a basic version that worked under Ogre 1.0. (Maybe also 1.2, have to check)

It is based on a cubemap to make a sphere out of 6 clipmaps, not on a single clipmap like the previous link.
It has no compression for texture and heightmaps. The heightmap is read on the fly from the harddisk with some very simple predictive preloading.
The texture has a big texture and a detail texture.
Because of the reading from disk there can appear some hickups when you approach the planet.
There are no rayscenequeries implemented.
It compiled under linux and VC6. But because the support for VC6 is now dropped and I don't have an other VC yet I'm don't knownot sure if it will compile on other win systems.

For the moment I don't think it is worth licensing it under a restrictive license because it is not production ready. So I'll release this version under the BSD license.
If I continue to work on it I will release the "full" version under LGPL or so.

Tuan - I don't think the plugin is userfriendly enough to make it an official addon yet. But I'll try to find some time to work on it. (No promises though) But I don't want to keep anyone from "upgrading" it if they want, so I'll make it available for download asap.

For the moment I'm checking if the version I have in front of me is my last version. And if it works alright. Then I'll upload it somewhere.

Post by **Assaf Raman** » Tue Oct 17, 2006 8:30 am

Curantil – thanks for your quick replay. I will be waiting for your code.

Tuan – As you proposed - I started to research compression algorithms that are able to decompress “region-of-interest” (ROI) queries at any power-of-two resolution.

I started with the article Hoppe wrote about: "Fast Progressive Image Coding without Wavelets"
http://ieeexplore.ieee.org/iel5/6760/18073/00838164.pdf

I didn't understand from the article how to do that.

I continued to search Google and found this patent:
http://www.patentstorm.us/patents/7095900.html

A method and apparatus for performing scalar quantization with a power of two step size is described. In one embodiment, the method comprises receiving a sequence of image data to compress and specifying scalar quantization with a power of two step size using three bit values to apply to the sequence of image data.

Seems the idea of power-of-two resolution decompression is worth a patent. Today I am going to search for any existing code that can do that.

Does anyone knows of an existing wavelet or not based existing code that can help? Or of someone has implemented Hoppe idea?

Post by **tuan kuranes** » Tue Oct 17, 2006 10:40 am

I started with the article Hoppe wrote about: "Fast Progressive Image Coding without Wavelets"

http://research.microsoft.com/%7Emalvar/ contains original paper and code (lapped orthogonal transform, LOT) from the author of PTC algo.
Might be of great help.

I continued to search Google and found this patent:

That Patent you posted is a about using wavelets to do this. algo used is without wavelets.

I didn't understand from the article how to do that.

Here's my understanding :

2 different things to handle, and to be developped in that order :

- "Pyramidal map" : use "level-1" to interpolate points of "level" uW
an interpolatory subdivision scheme specified in the paper. seems just a quadratic interpolant using a mask weight on each four point around in upper level:

Code: Select all

p = (–1/16 * p1, 9/16 * p2, 9/15 * p3, –1/16 * p4)

- Difference between interpolation and real value is stored and compressed (using PTC here.).

So getting a value at x,z is

Code: Select all

p(x,z, level) = getStoredCompressedDifference(level) + Interpolation(level - 1)

So you might better start wihtout compression of the "difference", just saving each level "difference" map into png files.
Only when you got that working, you can try to add PTC compression.

Post by **Assaf Raman** » Tue Oct 17, 2006 11:51 am

tuan - I will try to implement something like you describe. Thanks for the detailed answer.
I hope to get a simple demo running in the next few days.
I will try to build code that builds the diff levels, and save them as PNG.
Then code that read the PNGs and up sample to the original image.

curantil · Post by **curantil** » Tue Oct 17, 2006 12:08 pm

Ok, I have the code here : http://curantil.googlepages.com/planetscenemanager with a little explenation of how to use it.

I haven't tested if it works under Ogre 1.2, but it is the last version I have.

If you have questions about it feel free to ask.

Post by **Assaf Raman** » Tue Oct 17, 2006 5:16 pm

curantil:
I converted your code to OGRE v1.2 [Dagon] and

.

I also used some of the code that tuan created in the past.

Here is your code with

support:
http://www.upload2.net/page/download/to ... r.zip.html

Here are some screenshots:

You can use the screenshots and the code on the site you created.

curantil · Post by **curantil** » Tue Oct 17, 2006 5:49 pm

Wow splendid work Assaf!
I've put it on the site so everybody can get it.

When I've more time I'll check the convertor and so to see if it works