Project OgrePlanet - Slowly coming back to life

[HexiDave]

Hope that title isn't misleading, but I've been working on this project off and on for a while with varying degrees of success. I've recently hit a snag in the project, or what I perceive as one, and I've also promised to release this code for a while. I figure I'd kill two birds with one stone here and try to get both done at once.

The project is basically just building planets to use with Ogre. I use a quad-tree based method that breaks down heightmaps and then projects them onto a sphere. It works pretty well and is fairly fast even on my 2 GHz machine, 512 MB RAM and Radeon 9600.

There's a problem, though. Everything has gone very smoothly up until a few days ago. I was testing the application from a slightly different angle - I would usually build up the map instantly (skipping any sort of queue) and tearing it apart in the same fashion. It worked fine, but I recently added the queue and actually let it run as if it were a real application - that is, I let the LOD/Camera system take over. It worked great... until I started accumulating a lot of stuff into memory. I started noticing some weird things, like when I let the program randomly build further than I would let it go in a real test and it would get to maybe 200 MB of data in memory... then suddenly drop to 33 MB. Well that's odd... Next, and this is the killer for me, the quad-tree neighborhood breaks.

Now, I'd normally attribute that to a bug, however the whole system is designed that all the children are self-similar, meaning if you were to section off a child and it's children, it would look exactly the same as the Root node and it's children/neighbors. The problem that occurs changes and isn't repeatable to the exact same error every time. It randomly chooses which depth it occurs at, which node it occurs in, which STEP it occurs in and even which neighbor it looses track of.

Again, this could just be a bug and I'm not reproducing it correctly, but I also tested my system's RAM (after one of my sticks stopped working, I was now left with 512 MB from 1 GB) and found something that didn't please me: I had memory errors. Horray, more broken parts I can't afford to replace! So, now I'm left with another possibility: the code works fine, just like it should, but the memory is eating nodes somewhere and I can't actually track it because the physical hardware is causing the problems.

So, here's my proposal: the code is already LGPL, you can fiddle with it to your hearts content. I'd like to receive any major upgrades you do to the code and any bug fixes, if you'd be so kind. I'll be working on the code myself and will make regular updates as I can, but any feedback would be most welcome. The thing of it is, though, this bug may not be a bug - it may just be my craptastic computer. I'd like to know if YOU can reproduce these results on your end. The specifics are as follows:

1.) The code almost always bugs out when I make it build up many levels of the quad-tree, usually 10+ deep.
2.) It only really happens when I pull away from the ground in a hurry.
3.) The debug version will throw an error 75% of the time when it's using the BuildQueue::adjustLOD() function.
4.) If you look at the Terrain's QNode parent (pTerrain->getParent()) and look at it's neighbors (mNeighbors[0-3]) you'll see one of them is bad!
You'll see one listed as 0xfeeefeee or something like that and it won't be evaluable.

That's about it for that - if it turns out to be a bug, then I'll probably scrap the neighbor system and use another method I have in reserve. It's not quite as fun, but it should do the job (or kill me in the process.)

Here's a shot from the very unfinished system:


As you can see there, the LOD code is sort of working, but not all of the cases have been taken care of. This is also baffling me as the LOD code WORKS with the quad-tree neighbor system all the way down... it just seems to break at the weirdest times...

Anyways, here's the code and media - check the Readme file first as it'll tell you what you need to do. The only thing I forgot is:
You need to put that Front.raw file in your Release/Debug folder. That's the master map for that planet-face.

Download:
https://github.com/OGRECave/ogre-planet

Enjoy and don't throttle me for the lack of diagrams - they will come soon. Promise

[tuan kuranes]

Nice project !

Same segfault bug here.

Really seems to come from qnode::unsplit() not making neighbors correctly aware of the children deletion. Perhaps each current QNode neigbour should be responsible to update itself (parent or children) when a QNode is unsplitting, leading to a method like QNode::freeNeighbor(int, depth, qnode)

While I'm here, I'll drop some tips :

1) here's how to easily/hackily load a resource like a .raw file :

Code: Select all

FileInfoListPtr finfo =  ResourceGroupManager::getSingleton().findResourceFileInfo (
    "OgrePlanet", fileName);
FileInfoList::iterator it = finfo->begin();
if (it != finfo->end())
{
    std::ifstream rawFile;
    String completeFileName ((it)->archive->getName() + "/" + fileName);
    rawFile.open(completeFileName.c_str(),std::ios::in | std::ios::binary);    
    assert (rawFile.is_open ());
    mMap = new float[iWidth*iWidth];
    float* M = mMap;
    MUShort h = 0;
    for (int j=0;j<iWidth;j++)
    {
        for (int i=0;i<iWidth;i++)
        {
            rawFile.read((char*)&h,2);
            *M++ = h;
        }
    }
    rawFile.close();    
} // if (it != finfo->end())

2) Try using some assert.

3) Variable naming is confusin as variable beginning with a 'm' should really be class members, not local variables...

[tuan kuranes]

Bug indeed confirmed, if I recurse reset all neigbor of every single qnode that is equal to "this" inside the ~qnode delete, it works. (but doing a whole tree traversal at each node removal is ...).
So you know where the bug is and how to clean that, at least in a hacky waywhat to do.

Code: Select all

// new hacky method
void QNode::emptyNeighbor(QNode* node)
{
		for(int j=0;j<4;j++)
		{
			if (mNeighbors[j] == node)
			{
				mNeighbors[j] = 0;
				//break;
			}					
		}
		if (hasChildren())
		{
			for(int j=0;j<4;j++)
			{
				QNode * const cNode = getChild(j);
				if (cNode)
				{
					cNode->emptyNeighbor(node);
				}
			}
		}
};
// modified start of qnode.
QNode::~QNode()
{

	if (mParent)
	{
		QNode * pNode = mParent;
		while (pNode->getParent())
			pNode = pNode->getParent();
		pNode->emptyNeighbor(this);
	}
....

Then in release mode, at 600fps, some problem is rising about renderlevels in getIndexData but only when moving very, very fast. But that's a problem for later.

[HexiDave]

Well now THAT'S interesting... I appreciate the help there - I think I'll look a bit more closely at all this once I'm more awake. I'm actually kind of glad you had the segfault issue - it means that my computer will survive more Dev work

I do think you're right about letting nodes do the work themselves instead of parents/children. I had thought about that a few days ago, but that's when the bug started showing up and I figured I'd release the code and see if it was the PC rather than my shoddy code...

The naming thing was a big *doh* as I had originally written it all in one style and stole a lot of that code from my old prototype, which was basically where I had started up C++ after a few years of not touching it

I'll get the code re-arranged and I'll even do some diagrams if I can get it all fixed in short order - I'll toss a Wiki up as well.

Again, really appreciate the fix there - I don't know why I didn't think to look at that.

[Grom]

Wow HexiDave I was thinking writing a planet SM myself, yours looks damn cool! Tuan always impresses me, that guy is such a good programmer. I wonder if he's thinking of making PLSM3 planetary...

[HexiDave]

I'm... I'm so retarded...

Code: Select all

pNeighbor->getChild(QNPCheat[i][0])->setNeighbor(pThisNode,QNPCheat[k][0]);
pNeighbor->getChild(QNPCheat[i][1])->setNeighbor(pThisNode,QNPCheat[k][1]);

For those without the code:

Code: Select all

// Cheat for loops
static QPosition QNPCheat[4][2] = {
	{SW,SE}, 
	{NW,NE},
	{NE,SE},
	{NW,SW}
};

In that first part, the loop gives me "i", which is 0-3, which correlates to North, South, West and East. "k" is the reverse of "i", so North -> South, West -> East, etc. The idea was to skip a huge "if" section... Now, for those of you paying attention as much as I did when I wrote this code, you'll notice that I'm not setting the neighbor to North, South, East or West, I'm trying to set it to NW, SW, NE, SE... Same numbers (0-3) but it's completely wrong...

So, in short: I'm a retard. The code there in the unSplit() command should be:

Code: Select all

				pNeighbor->getChild(QNPCheat[i][0])->setNeighbor(pThisNode,k);
				pNeighbor->getChild(QNPCheat[i][1])->setNeighbor(pThisNode,k);

So far so good...

Edit: Whoops, didn't see another post.

He's more than welcome to integrate any of my code into PLSM3. It's mostly the same setup anyways, it just doesn't use the same basic structure (i.e. quad-trees, procedural content for the most part, etc.) The thing is, also, that my system isn't setup as a SceneManager, it's all MovableObject/Renderable inherited. I do the LOD checks manually instead of letting the SM do it.

So, ya, PLSM3 + Planet stuff; feel free to add it in.

[Praetor]

I think a collaboration is a good idea.

So far it looks good. The screenshot your posted is quite impressive. I believe this to be technological challenge many just cheat to get by, but having a proper solution should provide a lot of freedom.

[HexiDave]

Uploading shots & new code (and hopefully video/compiled demo) shortly, but I got the stitching working. Amazing what a few bug-fixes will do for personal morale, eh?

So, now you can zoom in and out and everything stitches up nicely. The LOD scheme still is a bit underhanded (just bases level on distance to camera), but that's on my ToDo list along with a few other things:

1.) Implant the Floating Origin system so you can have big planets without the jittery floating point garbage.

2.) Smooth out some of the shoddy "if"'s into loops (or unroll the loops I have) so people aren't wondering what the hell is wrong with me when they look over my code.

3.) Physics - going to put in a basic physics system so people can bounce around - I'll probably just start with ray-casting type thing and then move onto PhysX or a plug-in system where I show you where to put the main physics code (it'll be heavily integrated into the Floating Origin system.)

4.) Maybe chunked/continuous LOD such that the world doesn't pop like crazy. If you have any good resources on how error metrics would work for a spherical system, I'd appreciate it - even if it's for GeoMipMapping. I seem to have skipped over how all the error metric stuff worked...

5.) Texturing. I've gotta sit down on this one a bit - I have 2-3 methods that should work nicely without making the engine putz along on older systems. Right now I just have that simple Normal Mapping thing going with a 1D texture lookup. I'll be implementing height vs. slope lookups with texture splatting that will be geometry independent (i.e. all done with textures on GPU.)

6.) Horizon Culling - I've got a basic system I'll be placing in soon that should help when I have all 6 faces of the planet active. As of right now you're not hurting too much.

7.) I dunno, I have plenty of stuff lined up (atmospheric scattering, water, etc.), but I'm trying to focus on getting speed out of the thing.

If you have anything you want tacked onto the project, PM me or post here. No guaranties, but once I get cooking I should be able to open it up more to the community rather than just stuffing updates into posts. Maybe I'll qualify as an Ogre Addon project at some point.

Anyways, expect an edit shortly with some downloads.

[HexiDave]

Ok, here's the new source-code and a few more shots. After I took these shots I also realized that I should make a better 1D texture, so the default now has a more earth-colored coloration. It's all changeable in the material file.

Download code/media:
http://www.fileden.com/files/2006/7/10/ ... Planet.zip

Images:
http://www.fileden.com/files/2006/7/10/ ... /Curve.png
http://www.fileden.com/files/2006/7/10/ ... ground.png
http://www.fileden.com/files/2006/7/10/ ... wesome.png


As before, if you see bugs (or more importantly in this one, cracks in the ground) post here or PM me. I'm hunting for my copy of Fraps to take a video, but I really need to get that BuildQueue on a timer again first (otherwise the whole PC chokes.) I'll also try to get a demo download for you to mess with (need to make a few adjustments to how the controls are so you can actually get closer to the terrain.)

You'll notice that a lot of areas are much larger than they need to be - that'll change - it's just the way the vertices are adjusted at the moment, but I can even have it so you can declare whole sections to be flat and the geometry will stitch the edges properly still.

[tuan kuranes]

@PLSM3 related: I should use a ghost account . I'm glad there is multiple Terrain, Planetary and/or paging implementations in Ogre. I even think what ogre misses is more Scene Managers. About PLSM3, I'm near completion not really a time for enlarging the team. I work hard for this and even had to drop plsm2 and ogreode forum support for that (sorry. the good thing is that people are helping over there, thanks). After first release, any contributions will be welcome. enough OT.

@HexiDave: Glad you succeed.

As before, if you see bugs

Same bug in release mode as noted in above post when moving very fast.

I'll also try to get a demo download for you to mess with

try to make use of code above so that the .raw file can go in the media folder. Some real textures and heightmap (mars, phobos, moon, etc...) are available on the net or even on my website.
texture are much better than vertex based coloring when using LOD. Note that you can use 16 bit png instead of raw.

need to make a few adjustments to how the controls are so you can actually get closer to the terrain

You'd better implement a getHeight to make camera stick to the ground like in TSM (you'll need one anyway for physics.). hacking some control system is very hard (PLSM2 one is just a mutliplier one (move = 1, shift + move = 10, ctrl + move = 100, shift+ctrl+move = 1000 or something like that.))

How did you make that Hole?

[HexiDave]

About that bug, I haven't gotten it yet and I'm zooming in and out as fast as I can... Still, there's only a few places that LOD stuff happens, so I'll check the math.

The raw->PNG thing, right. I had actually forgotten about that I was mostly sticking with RAWs as that's what it was spit out as (I didn't have a PNG output code at the time.) It's a non-issue for me as I can hack that in shortly. I'll get right on it.

As for textures - I know about the per-vertex coloring. If you think about how many levels there are, you'll see that any texture would stretch beyond recognition used normally. I have a system I'm building now that will basically build textures that will be filtered down every few levels. It will use the created heightmaps and it will be added to as you go down. Basically imagine a blank texture that would cover the top-left quadrant and the next 3-4 levels of nodes below that. As I build heightmaps that would be in that texture, I add it to the texture itself. Now I can keep from making tons of little per-tile textures. I'll also be able to encode a variance map (done with some noise) to help blend textures when splatted.

Now that control system - it's not going to work exactly the same, I think, but I know what you mean. My first goal for controls is to not use the EnvMapping file and go build my own ExampleApplication-style demo program. Not a big issue, but I still don't feel like doing it hehe...

That hole in the ground was just from the terrain synthesis. I was just making note of how some of the features can look. That's also with the old roughness factor in the synthesis function as well as before I adjusted the radius scale as well. The terrain looks a lot better now (no more pointy mountains) with a lot more rolling features.

I just got up so I'm going to finish waking up before I get some code done. I'm debating what I want to work on first, but I'll definitely get the Timer attached to the BuildQueue system such that the computer doesn't choke when building tiles. I also really want to put that Floating Origin system in to place so I can expand the size of the planet so the camera can get in low. That texture system is going to take a little bit of testing with since I can't just plug bits and pieces in off-hand, I need a lot of little offset factors I need to test (I always goof on those when I rush.)

Anyways, I think I'll soon be able to add the other sides of the planet and do horizon culling, so it should start looking like a planet. I'll try to get the compiled demo out today and maybe a video. Maybe I'll look into SVN or something.

[Chimelion]

HexiDave, thank you for sharing your beautiful OgrePlanet! I have started playing with your code to see what I can learn and hope to contribute something back, with experiments growing out of PLSM2.

OgrePlanet, Chaster's portal work, Sinbad's Tindalos, tuan kuranes work on PLSM3, and others! Ogre3D, go team!

Currently, I'm hacking on my crude implementation of water bodies, which I feel might need to be a specialization of actual height-mapped dynamic terrain, rather than a simple plane with a shader on it--using an LOD strategy, the plane-and-shader seems ok for the distance, but I want something more compelling right up close to the camera, especially at the interface between a land mass and the body of water, so that's what I'm experimenting with.

Have you thought about water implementations much?

[HexiDave]

Well I have to put the project on hold slightly for a few weeks (go go gadget Real Life), but you could use another quad-tree sphere with fewer verts per patch (maybe 9x9 or something) and just let the pixel shader fake the depth for the most part with some waves in the vertex shader. Wouldn't do much damage to the frame-rate and could be interchanged with ocean areas (i.e. if a section of ground is completely submerged by the ocean, stop displaying it until you can see it through "sea-fog.")

You also won't have to deal with heightmaps - just straight cube->sphere projections. The quad-tree system itself takes up almost nothing as far as memory goes, so as long as you're using that video RAM, might as well not bother writing a ton of specialized code while you're at it.

I have to take another look at how I'm doing the quad-tree splitting/unsplitting as some funny (ok, not "haha" funny, but "i'm going to stab someone" funny) bugs keep popping up where they shouldn't be.

I've got to get a job and I've got an interview in a few days, so hopefully I can make it a routine and get back to working on this soon. If you need help with your implementations, just let me know here or PM me and I'll get back when I can.

[Chimelion]

Thank you for the tips, good food for thought.

And good luck on your job interview!

[pekar]

Nice work, it's nice to finally see it in action, and get to peek at what makes it tick. Any plans to make for this in the future?

[HexiDave]

I've got all those outlined things above still planned as well as other things that a planet would need, but it's all on hiatus until I can get a job going. As said, I think there's a bug in how the system works now because for some damned reason I just can't plug it into a queue system, such that I can hold off on splitting a tile until I've got the children built. The quad-tree neighborhood always looses cohesion somewhere.

It's such a diabolical system that it's hard to debug it properly (i.e. visualizing transitions of a few dozen events that take place at once.) If worse comes to worse, I'll create a big visualization system to point out what neighbors each terrain patch has and create transition effects to show what it's going to be pointing to, what depth it's at, what neighbors it affects when splitting, etc. I want to do that anyways, but time is out right now.

If you need help with anything, just post here or PM me and I'll respond.

[MadLion]

Hi,

very interesting project. ^^

How did you create the raw heightmap file?

[HexiDave]

The one that's included with the project is from a modified example from Libnoise - the Complex Planet example to be more specific. Basically I adjusted the creation settings I liked, fed it projected vertices from a cube-face (you'll have to read a bit of what I did with the system to understand this part) and just pulled the values out of libnoise and put them into a heightmap.

I'm slowly grinding my gears to get the project up and running again, but my employment is still up in the air. I need to solve the issue with the quad tree where it degenerates the neighbor links at some levels, which would involve me creating a rather powerful debug visualizer for the quadtree. To picture how it's going to work, imagine a square, then imagine another 4 below it that would fit into the corners as if it were a copy of the above square and split into 4 pieces. Keep doing that for each subdivision you do in the quad tree. Now, draw a line from each edge to the center of the target neighbor for that direction. You now have the basic system for the debug visualizer. I just need to know where and when the neighborhood breaks down and then I can find a solution.

[betajaen]

Wow. I would love to see one of the physics engines to get their grubby little paws on it. Combine it with Caleum and Wow-sa!

[HexiDave]

Oh, it's on my to-do list and I had a small test implementation, but I REALLY have to get that stupid neighborhood system to work properly. It works, but it just breaks down - very strange, so it's probably a funky recursion problem (I've even done up to about 10 levels deep on paper and it works).

[MadLion]

Thanks HexiDave.
I'll have a look on it.

[pin_]

Hey HexiDave
I'd like to ask what's the current stage of the project. Is is in a usable state?
Also what are the licensing terms.

Thanks
Calin

[Cutter]

According to the readme, it is LGPL.

Have you made any progress?

[HexiDave]

I haven't touched it lately, no. I'm very short on time for the moment and I'm focused on the Simple Paged Terrain system over in Showcase. I WILL be coming back to this in time, but for now it's on hiatus due to the fact that my PC is at Old Yeller stage and I can't reproduce a bug that happens when I move around fast enough.

It is LGPL so you're welcome to do what you like with it, but if you make changes I'd like to see what you did with it as I can always use it later when I get back into it.

[petrocket]

I don't suppose you've seen this before?

http://www.sponeil.net/

i was trying to port this to ogre when i did a search and saw your post.

and this is his explanation:
http://www.gamasutra.com/features/20060 ... l_01.shtml

of course everything at the infinity dev blog is real helpful:
http://www.gamedev.net/community/forums ... ?jn=263350[/url]