New Terrain Early Shots

[jacmoe]

Rel-with-deb-info FTW.

[Ter13]

I just got this bad boy built finally, and I'm gonna give it a shot. Good going Sinbad! Seems like all you have left is paging.

[sinbad]

First cut of the working paging system is in. However, don't get too excited yet, it clearly needs more tuning because the performance when a new page is brought in is not nearly good enough yet. I'm not that surprised, there are a number of things I can do try to improve that. We might end up with an RC1 where the performance isn't great and refine it as we go forward.

For info, here's how you might set up the terrain paging, assuming that you've already got saved terrain files for all the pages in question and the setup is the same as the terrain sample:

Code: Select all

		mTerrainGroup = OGRE_NEW TerrainGroup(mSceneMgr, Terrain::ALIGN_X_Z, TERRAIN_SIZE, TERRAIN_WORLD_SIZE);
		mTerrainGroup->setFilenameConvention(TERRAIN_FILE_PREFIX, TERRAIN_FILE_SUFFIX);
		mTerrainGroup->setOrigin(mTerrainPos);

		configureTerrainDefaults(l);
		// Paging setup
		mPageManager = OGRE_NEW PageManager();
		mPageManager->addCamera(mCamera);
		mTerrainPaging = OGRE_NEW TerrainPaging(mPageManager);
		PagedWorld* world = mPageManager->createWorld();
		mTerrainPaging->createWorldSection(world, mTerrainGroup, 11000, 12000, -2, -2, 2, 2);

That last call basically does all the work - it creates a world section which pages according to the Grid2DPageStrategy, aligned with the TerrainGroup settings, and sets the "load radius" to 11000, the "hold radius" to 12000 and sets a limit on the grid extent to (-2,2) in both directions - essentially the entire dataset is a 5x5 grid and I've already saved terrain files for every cell in this grid.

You could actually add other things to that world section to page other stuff in/out too, but that's for the future. Firstly I have to get the terrain paging smoothly enough to be usable, because right now there's way too much of a hiccup.

[sinbad]

For those on lower-end cards but which are still SM3-compatible (like Daniel's 6600), please try the latest code. I've added dynamic branching to hopefully reduce shader instructions and texture accesses by skipping many calculations where the blend value for that layer is 0. It doesn't really seem to make much difference on my machine but then I'm not pixel shader bound here (400fps normally, 600fps when looking directly at the ground!) so I'm hoping it makes a difference for older SM3 cards.

[aguru]

Great work on the terrain paging!

I just found out some more about that graphics bug I was writing about a few posts up. It's the colourmap that causes these. I just uncommented the code that sets up the colourmap in Terrain.h, line 483. I used a 512x512 jpeg.

Some observations:
Without colourmap, and once the background calulations are finished, I get a framerate of about 1200 when hovering above the terrain (camera high enough for it to render composite maps only). (yay!)
With colourmap enabled, calculations finished and looking from the same spot, fps is only 180. Also, only one terrain instance displays the colourmap properly. When restarting with the generated dats, everything is screwed up (interlaced lines instead of the colourmap in the working terrain instance, but fps is as expected.

Hope some of this is of help

[sinbad]

Ok, I haven't re-tested the colourmap since the TerrainGroup support in fact, so I'll put that on my list.

[sinbad]

Colourmap save/load is fixed, was a small copy & paste bug in the saving code.

Fps difference remains when you use a colourmap - on my machine 120fps when calculated in this run, 400fps when loaded from a file (no difference if you don't use a colourmap). I'm investigating.

[sinbad]

FPS difference is fixed, there was a lingering 'dirty' setting on the material params that would never get reset if you altered the global colour map option after construction.

[simply]

The new terrain system is just awesome. I've implemented it in seconds and everything worked without a hassle. Keep up the good work!

[rubasurex]

I just tried building and received a "Cannot open include file: 'OgrePagedWorldSection.h' error from OgreTerrainPaging.h. In CMake, I only ticked to build the Terrain component. I'm not using paging in my project. Do I need to use the Paging component as well now in order to continue using this terrain?

[sinbad]

I just tried building and received a "Cannot open include file: 'OgrePagedWorldSection.h' error from OgreTerrainPaging.h. In CMake, I only ticked to build the Terrain component. I'm not using paging in my project. Do I need to use the Paging component as well now in order to continue using this terrain?

Ah yes, there's a build-time dependency there now, sorry. I can put in some #ifdefs to block it out when the paging system isn't enabled though since the majority of classes don't refer to paging. Logging a bug so I don't forget - for now just tick the paging component too.
[EDIT]Nm, fixed. I'd already included some conditional sections in the CMake config I just hadn't used them in the latest updates. Done now.

[DanielSefton]

For those on lower-end cards but which are still SM3-compatible (like Daniel's 6600), please try the latest code. I've added dynamic branching to hopefully reduce shader instructions and texture accesses by skipping many calculations where the blend value for that layer is 0. It doesn't really seem to make much difference on my machine but then I'm not pixel shader bound here (400fps normally, 600fps when looking directly at the ground!) so I'm hoping it makes a difference for older SM3 cards.

Unfortunately I can't see much of a difference either, but again, it's hard to tell because it varies so greatly. But every little helps.

I encountered more problems when I upgraded to the latest revision:

- I randomly get a crash on exit.
Edit: here's the stacktrace:

Code: Select all

	Client.exe!Ogre::TerrainMaterialGenerator::~TerrainMaterialGenerator()  + 0x15c bytes	C++
 	Client.exe!Ogre::TerrainMaterialGeneratorA::`scalar deleting destructor'()  + 0xe bytes	C++
>	Client.exe!Ogre::SharedPtr<Ogre::TerrainMaterialGenerator>::destroy()  Line 229 + 0x23 bytes	C++
 	Client.exe!Ogre::SharedPtr<Ogre::TerrainMaterialGenerator>::release()  Line 217	C++

- The scene fog on the terrain has stopped working. If I destroy and recreate the terrain, the fog appears, but it's inversed.

- I get a crash when I destroy the terrain, then recreate it and call renderOneFrame() or renderWindow->update() early during it's initialization. (Something you'll probably never reproduce; it's for updating my load bar)

- getHeightAtWorldPosition() doesn't work properly if terrainGroup->setOrigin() y is more than 0.

[Xplodwild]

Great job again.

I'd like to know, is there a method to know if terrains are loaded/in loading or not ? Especially when loading terrains in threaded mode, since I have to generate heightfield for physics.

[DanielSefton]

Take a look at the sample in SVN, you can use isDerivedDataUpdateInProgress() for a Terrain or TerrainGroup, in frameRenderingQueued():

Code: Select all

if (mTerrain->isDerivedDataUpdateInProgress())
{
	// Not fully loaded

	if (mTerrainsImported)
	{
		// Building terrain
	}
	else
	{
		// Updating textures
	}
}
else
{
	// Terrain loaded!
}

[Xplodwild]

I tested it some days ago, and isDerivedDataUpdateInProgress() returned false after calling loadAllTerrains(false);. So I ended up with a crash when calculating physics heightfield. Right now I'm updating from latest SVN, maybe sinbad changed something, and I'll re-test.
Does it have to be absolutely in frameRenderingQueued ?

[sinbad]

Yeah the isDerived() call is about derived information like lightmaps etc because of a change, that's not the same thing. Call isLoaded() on a given instance to see if the main data is loaded yet. I have load events on my TODO.

I just implemented index buffer pooling which gave me a 10% performance boost here (as well as reducing GPU memory usage).

Re shutdown crash: this is probably because the TextureManager (or Ogre Root) is getting destroyed before the material generator's TexturePtr. I changed it recently while looking for another bug to a real TexturePtr, should probably change it back; I thought the problem wasn't there anymore (and I haven't seen it since) but I can see the potential.

Re fog: still works fine for me.

I'll look into the other issues when I get time, I wasn't seeing them here.

[Xplodwild]

Also, is it normal that when you disable normalmaps texture, you end up having no lighting and no lightmap at all ?

[sinbad]

If you mean calling Terrain::_setNormalMapRequired(false), then yes, because you shouldn't be calling that. As described in the docs, it's the TerrainMaterialGenerator's job to decide what it needs from the Terrain instance to render correctly and it calls this to tell the terrain it needs to generate a terrain-wide normal map to perform the lighting. If you wrote an alternative TerrainMaterialGenerator which didn't need this then it would request different options. But TerrainMaterialGeneratorA needs the global normal map and will not render correctly without it. This normal map is just the per-vertex normals encoded as a texture, and exists because per-vertex lighting doesn't look good on variable LOD. In fact it's only used relatively close-in and for generating the composite maps - but still you would notice the lighting change in close to medium distance LOD transitions if vertex lighting was used instead.

If you want to turn off normal mapping (as in, the normal maps per layer), you should call TerrainMaterialGeneratorA::setLayerNormalMappingEnabled. This does not prevent lighting, just the per-layer normal map information.

[Xplodwild]

I thought that removing normal textures (ie. not adding them to the layer declarations) would disable it automatically.

[DanielSefton]

Right, linear fog works fine. It's exponential fog that's inversed.

Crash on exit bug is fixed. Also, I don't seem to get the crash when updating the render window during initialisation anymore, which is nice.

[sinbad]

I thought that removing normal textures (ie. not adding them to the layer declarations) would disable it automatically.

No, because the layer declaration must also be compatible with the material generator. That's actually why the material generator provides it. Mostly the declaration is there to help tools know what components they have to supply and where, and to record what's contained in the terrain data so that you know how to re-process the data if you need to adapt it to a different approach. It's not designed to dynamically adapt to any declaration (too many variations).

Right, linear fog works fine. It's exponential fog that's inversed.

Yes, only linear fog is supported right now. Can be enhanced in the material generator but I have too much on my plate right now to worry about it.

Crash on exit bug is fixed. Also, I don't seem to get the crash when updating the render window during initialisation anymore, which is nice.

Both probably related to the static texture variable issue, which I fixed.

[sinbad]

Dynamic shadow support is being added to the terrain. Both simple projective (colour) shadow textures and depth shadows are supported, here's a couple of shots:

Simple non-depth unfiltered shadows

terrainshadows2.jpg (121.36 KiB) Viewed 6525 times

Soft depth shadows

terrainshadows3.jpg (167.58 KiB) Viewed 6525 times

Depth shadows in particular aren't perfect yet, they're mostly working but I'm getting some artefacts when close to the light direction, which may be a corner case in the scene_depth_range / shadow_scene_depth range or a LiSPSM issue, I'm not sure yet. Projected shadows have a bit too hard a cut-off at the far distance (or low-lod composite map render distance - by default the low lod distance render doesn't include dynamic shadows although that's an available option). I'll look into all these things after Christmas.

If you want to try it out for yourself, just edit the terrain sample's Terrain.h and uncomment:

Code: Select all

//#define SHADOWS
//#define DEPTH_SHADOWS

The first one activates shadows in general (just projected shadows by default) the second upgrades that to filtered depth shadows as well. Bear in mind that the latter is quite expensive. I will probably turn these into dynamic GUI options later but I'm flat out of time right now.

That's it for me for a few days at least, hope everyone has a great Christmas.

[Xplodwild]

Hi,

I tried it, it works well in the sample, but when I try to add it in my game, I get some strange colors : Textures becomes simple colors, and I still haven't any shadow:

pssmfail.JPG (29.17 KiB) Viewed 6432 times

I copied the same sample code as Terrain sample have, call it before creating terrain (I tried after, same result). I have one directional light in my scene. What's going wrong ?

And happy christmas for all.

[sinbad]

Need more info - hardware, DX or GL, any shader compile errors in the log, whether this is colour or depth shadows etc. I've only tested this on my 9800 so far (Dx9 & GL), it's still early code.

[Xplodwild]

Here's things I have in the log, I don't know if it's related.

Code: Select all

16:24:11: Terrain created; size=1025 minBatch=65 maxBatch=129 treeDepth=4 lodLevels=5 leafLods=2
16:24:12: Texture: spot_shadow_fade.png: Loading 1 faces(PF_R8G8B8,128x128x1) with  hardware generated mipmaps from Image. Internal format is PF_X8R8G8B8,128x128x1.
16:24:12: OGRE EXCEPTION(7:InternalErrorException): Unable to compile Cg program OgreTerrain/1465861243/sm2/vp/hlod: CG ERROR : The compile returned an error.
(22) : error C5102: output semantic attribute "TEXCOORD" has too big of a numeric index (8)
 in CgProgram::loadFromSource at ..\..\..\PlugIns\CgProgramManager\src\OgreCgProgramManagerDll.cpp (line 65)
16:24:12: High-level program OgreTerrain/1465861243/sm2/vp/hlod encountered an error during loading and is thus not supported.
OGRE EXCEPTION(7:InternalErrorException): Unable to compile Cg program OgreTerrain/1465861243/sm2/vp/hlod: CG ERROR : The compile returned an error.
(22) : error C5102: output semantic attribute "TEXCOORD" has too big of a numeric index (8)
 in CgProgram::loadFromSource at ..\..\..\PlugIns\CgProgramManager\src\OgreCgProgramManagerDll.cpp (line 65)
16:24:12: OGRE EXCEPTION(7:InternalErrorException): Unable to compile Cg program OgreTerrain/1465861243/sm2/fp/hlod: CG ERROR : The compile returned an error.
(73) : error C5102: input semantic attribute "TEXUNIT" has too big of a numeric index (16)
(75) : error C5102: input semantic attribute "TEXUNIT" has too big of a numeric index (17)
(77) : error C5102: input semantic attribute "TEXUNIT" has too big of a numeric index (18)
 in CgProgram::loadFromSource at ..\..\..\PlugIns\CgProgramManager\src\OgreCgProgramManagerDll.cpp (line 65)
16:24:12: High-level program OgreTerrain/1465861243/sm2/fp/hlod encountered an error during loading and is thus not supported.
OGRE EXCEPTION(7:InternalErrorException): Unable to compile Cg program OgreTerrain/1465861243/sm2/fp/hlod: CG ERROR : The compile returned an error.
(73) : error C5102: input semantic attribute "TEXUNIT" has too big of a numeric index (16)
(75) : error C5102: input semantic attribute "TEXUNIT" has too big of a numeric index (17)
(77) : error C5102: input semantic attribute "TEXUNIT" has too big of a numeric index (18)
 in CgProgram::loadFromSource at ..\..\..\PlugIns\CgProgramManager\src\OgreCgProgramManagerDll.cpp (line 65)

Hardware is GeForce 7600GT, running on Dx9. The terrain sample works fine btw. I tried depth shadows and colour shadows, the result is exactly the same.