New Terrain Early Shots

[Jabberwocky]

Shame I was expecting some sort of mad accent tinged with a bit of Yorkshire, but you sound fairly normal in a neutral Southern sort of way

Sinbad had to use a voice actor for that video, since his real accent is quite hard to understand. Here's a sample of the original, unedited footage with Sinbad's real voice.

[DanielSefton]

[mikeInside]

thanks for taking the time out to make a video, was very nice.

[Nauk]

Shame I was expecting some sort of mad accent tinged with a bit of Yorkshire, but you sound fairly normal in a neutral Southern sort of way

Sinbad had to use a voice actor for that video, since his real accent is quite hard to understand. Here's a sample of the original, unedited footage with Sinbad's real voice.

LOL

[sinbad]

Had my sound switched off and wondered why there was no movement until I realised Steve was giving a commentary. Is that a first Steve? Shame I was expecting some sort of mad accent tinged with a bit of Yorkshire, but you sound fairly normal in a neutral Southern sort of way

Haha - yeah, this was a first. Like most people I think my own voice sounds ridiculous but I figured I'd try doing it - this was mostly inspired by Omniter's videos over the GSoC which I thought were a great way to inform people. If people found it useful I might do some more.

And yeah, I'm about as far south as you can go and still be in British accent country. Well, except the Falklands I guess I suppose my accent is mostly a local Channel Islands one, maybe mixed with occasional slight Cornish twangs from my Dad (which my wife ribs me about sometimes). My Mum was only in Yorkshire until she was 2 so I didn't get so much of that, although my Granddad stayed a broad Yorkie for 40 years after he moved down south! So at least I can understand what people are saying in that part of the country

[Jabberwocky]

If people found it useful I might do some more.

Where there's something visual to show off, a narrated video is for sure the way to go, and definitely appreciated.

[jacmoe]

I enjoyed the video!
Was nice to hear you talking while you showed off the terrain.
Would be great to have a series of videos of Steve in Ogre Land.

[reptor]

Oh, the video was good. And the voice too. Very good. I could understand it very well. It's good you speak in a clear manner because you have an international audience here. I know not all do - when I flew from Finland to the UK using Ryanair, the guy speaking to the microphone in the airplane was probably Irish and I had trouble understanding him at first. He had a really strong accent. But I did eventually start to understand most of what he was saying, but it required quite a lot of effort. The ladies there who also spoke to the same microphone were speaking in a much clearer accent... in fact the guy looked like he didn't even care if we understood or not bored of his job probably.

Thumbs up for doing videos like that.

[DanielSefton]

Just checking out the new terrain. I don't know if this has been mentioned before, but how do you reduce/remove this really intensive specular effect?



Also, is it possible to load .raw heightmaps?

[aguru]

I don't know if it is possible to tell the terrain Material about specularity but Caelum's speculariy is pretty strong, you can try and reduce it

Code: Select all

mCaelumSystem->getSun()->setSpecularMultiplier(ColourValue(.1, .1, .1));

Loading raw heightmaps is a matter of changing one line, like this:

Code: Select all

Image img;
DataStreamPtr stream = ResourceGroupManager::getSingleton().openResource("heightmap.r32", "Terrain" );
//img.load("heightmap.png", "Terrain");
img.loadRawData( stream, 1025, 1025, PF_FLOAT32_R );
(...)

Happy experimenting

[DanielSefton]

Cheers

Though it doesn't like my 16 bit (PF_FLOAT16_R) raw heightmap. The terrain is upside down, and it stretches upwards into infinity.

[haffax]

Cheers

Though it doesn't like my 16 bit (PF_FLOAT16_R) raw heightmap. The terrain is upside down, and it stretches upwards into infinity.

Well, it has aesthetic power nevertheless.

[DanielSefton]

Well, it has aesthetic power nevertheless.

lol yeah, it looked like some kind of space lift.

Here's what it looks like without the mess: (I'm more pleased it works with DOF than anything else )



Looks ok considering I just tossed it in. Gives it a very different look/feel compared to MyRddin. I noticed, however, that it affects performance even more than MyRddin's terrain by about 10FPS, which is surprising. Waiting for paging support now.

[sinbad]

Regarding specular, this is controlled on a per-pixel basis by your specular map, which is usually the alpha channel in your diffuse map with the standard setup. If you want less specular, reduce the specular coefficient in that map. But yeah, Caelum defaults to some overbright settings, I almost always have to scale it back whenever I use it since the defaults are way over 1.0.

Did you try just feeding a direct float* in, or did you use a PF_FLOAT16_R image? I don't think I've tried the latter yet, if it's that, what image format did you use? I could probably use an example.

Performance wise, it really depends on what the settings are. Did your Myrddin version have parallax mapping, normal mapping and specular mapping on every layer? If not, you can turn those off for a better comparison, see TerrainMaterialGeneratorA::SM2Profile - it's the default so you can grab it via a cast from TerrainGlobalOptions or instantiate one of your own, set the options and pass it to the options yourself. Also, make sure you're using the same pixel error metric for comparison. I've never used Myrddin so I don't know how the new terrain stacks up.

[sinbad]

Loading raw heightmaps is a matter of changing one line, like this:

Code: Select all

Image img;
DataStreamPtr stream = ResourceGroupManager::getSingleton().openResource("heightmap.r32", "Terrain" );
//img.load("heightmap.png", "Terrain");
img.loadRawData( stream, 1025, 1025, PF_FLOAT32_R );
(...)

Well, you can do it this way, but it's far better to do it this way:

Code: Select all

Terrain::ImportData imp;
imp.terrainSize = 513;
imp.worldSize = 12000;
imp.inputScale = 1.0;
imp.inputFloat = pRawFloats;
...
terrain->prepare(imp);
...

Basically, the new terrain doesn't need to use Image to load the height data, that's just a convenience. It far prefers it if you just give it a list of unscaled floats!

Of course in production you would save the terrain data and just load it direct from the packed file anyway.

[DanielSefton]

Did you try just feeding a direct float* in, or did you use a PF_FLOAT16_R image? I don't think I've tried the latter yet, if it's that, what image format did you use? I could probably use an example.

Well it's simply a 2048x2048 raw heighmap generated from L3DT, 16 bits little-endian encoded, the same format I used for MyRddin. I haven't fed the floats in yet, I'll try that. Would be nice to have an example; surely I'm not the only one wanting to load a heightmap from L3DT.

This is how MyRddin loads it, if it's any use at all:

Code: Select all

#if OGRE_ENDIAN == OGRE_ENDIAN_LITTLE
		if (!bFlip)
			page->mpHeightMap.setNull();
#endif

        // Load data
        MemoryDataStreamPtr rawData;
        rawData.setNull();
        DataStreamPtr stream = 
            ResourceGroupManager::getSingleton().openResource(
                sFileName, ResourceGroupManager::getSingleton().getWorldResourceGroupName());
        rawData = MemoryDataStreamPtr(OGRE_NEW MemoryDataStream(sFileName, stream));

        // Validate size
        size_t numBytes = mPageSize * mPageSize * 2;
        if (rawData->size() != numBytes)
        {
            shutdown();
            OGRE_EXCEPT(Exception::ERR_INVALIDPARAMS, 
                "RAW size (" + StringConverter::toString(rawData->size()) + 
                ") does not agree with configuration settings.", 
                "HeightmapTerrainPageSource::loadHeightmap");
        }

        // Convert the image data to unscaled floats
#if OGRE_ENDIAN == OGRE_ENDIAN_LITTLE
		if (bFlip)
		{
#endif
			const ushort* pOrigSrc, *pSrc;
			assert(!page->mpHeightMap.isNull());
			ushort* pDest = ((ushort *)page->mpHeightMap->getPtr());
	        
			pOrigSrc = (const ushort *)rawData->getPtr();

			// Read the data
			pSrc = (ushort *)rawData->getPtr();
			for (ulong j = 0; j < mPageSize; ++j)
			{
				if (bFlip)
					// Work backwards 
					pSrc = pOrigSrc + (mPageSize * (mPageSize - j - 1));
				for (ulong i = 0; i < mPageSize; ++i)
				{
					//always get as LITTLEENDIAN
	#if OGRE_ENDIAN == OGRE_ENDIAN_LITTLE
					*pDest++ = *((ushort *)pSrc);
	#else
					*pDest++ = ((ushort)(*(((uchar *)pSrc)+1))) | (((ushort)(*((uchar *)pSrc)))<<8);
	#endif
					pSrc++;
				}
			}
#if OGRE_ENDIAN == OGRE_ENDIAN_LITTLE
		}
		else
			page->mpHeightMap = rawData;
#endif

		//free temporary storage
		rawData.setNull();

Did your Myrddin version have parallax mapping, normal mapping and specular mapping on every layer? If not, you can turn those off for a better comparison, see TerrainMaterialGeneratorA::SM2Profile - it's the default so you can grab it via a cast from TerrainGlobalOptions or instantiate one of your own, set the options and pass it to the options yourself. Also, make sure you're using the same pixel error metric for comparison. I've never used Myrddin so I don't know how the new terrain stacks up.

I had normal and specular mapping with MyRddin on 3 layers. I'll try that, thanks.

This may seem obvious, but the FPS is impacted according to how much of the terrain is in view of the camera. If I don't look at the terrain, I get 50 FPS, if I look at a big chunk of the terrain it drops to 10 FPS, which is quite significant. So you think that's to do with the materials, not the geometry? When I don't look at the terrain, batch count is 7 and triangles is 30k. When I look at the terrain, batch count is 30 and triangles are 130k. MyRddin is basically the same as the old octree terrain but with paging, and I got a consistent 25 FPS with that. Bearing in mind I have an NVidia 6600.

[Praetor]

Can't say for sure but I really think it is pixel shader bound. That means the more of the terrain is in view at a time, the more pixels are being shaded with the terrain pixel shader.

[sinbad]

It may be - although in your tests there it's also increasing the vertex count a lot too and the 6600 is not that fast. I already deal with shader overhead and fillrate in the mid-far distance by using a composite map, so anything beyond the composite map distance (configurable) is using just a simple single texture to render with all the lighting and blending pre-baked. But close up it's always using the full dynamic shader pipeline. The only way to tell is to try varying the parameters and see which one affects the frame rate - set the composite map distance (TerrainGlobalOptions::setCompositeMapDistance - before creating the terrain) to closer and see if that improves things, and see if that makes things faster. Also try disabling the parallax mapping. Then separately, try increasing the pixel error metric. Which ones affect the frame rate will tell you where the bottleneck is. It should be possible to get it to run well, on my machines (9800 and HD2600) the new terrain is faster than the old TerrainSceneManager despite looking much better.

If your floats are just in a flat file, it's easy:

Code: Select all

		MemoryDataStream rawFloats(ResourceGroupManager::getSingleton().openResource(filename, groupName));
		imp.inputFloat = (float*)rawFloats.getPtr();

Be aware though of endian issues when you're using raw files like this. This file will only work if saved / loaded on machines of the same endianness (this applies to Myrddin too).

Basically we don't do this via a filename because it's far more flexible just to accept packaged data like this, the float data can come from anywhere. Also, you can load / construct / modify it in a background thread and even call prepare() in a background thread so the entire terrain can be bootstrapped without blocking. Again, I would expect you only to do it this way at import time, the prepare(stream)/save/load functionality of Terrain is what you'd use after that.

[PaddyWagon]

The only way to tell is to try varying the parameters and see which one affects the frame rate

Graphics perf is best instrumented and not twiddled with and guessed at, IMO. While a quick and dirty test will let you know that "something" changes you still have no idea whether it's vertex buffer copies, texture swaps, shader complexity, culling flaws, inefficient driver pipeline, batch counts, overdraw, any number of grue lurking out there in the dark.

Worst ones I've ever dealt with are driver version dependent, second was one where the dynamic tesselation went asymptotic but only when facing one direction 'cause the coder screwed up the math We thought it was a culling issue and then a sorting problem, lots of wasted time on that.

Pix. Nvperfhud. Intels' GPA. I'm sure there is an OpenGL out there somewhere. VTune. etc... all are a great help.

http://msdn.microsoft.com/en-us/library ... 85%29.aspx
http://developer.nvidia.com/object/nvperfhud_home.html
http://software.intel.com/en-us/articles/intel-gpa/

[DanielSefton]

The geometry is everywhere when loading it with raw floats

Anyway, the performance hit is definitely to do with the material. When the composite map covers the whole terrain, the FPS skyrockets. Though that's not much use until paging is implemented, with terrain in the far distance I guess.

How do you get the composite map to work with dynamic lighting? Setting Caelum's ambient and diffuse doesn't work, even per-frame. It's much lighter than the rest of the terrain.

Disabling parallax mapping increased average FPS by about 8. So now looking at a big chunk of the terrain averages at 10-20 FPS.

[reptor]

That's a very low FPS if you have only the graphics running. It's way too low...

[Praetor]

Wow that is kind of low. What kind of hardware are you running? Are there are other effects like post processing?

[DanielSefton]

It's just the terrain. I get 100+ FPS without it.

That's why I develop on an NVidia GeForce 6600, because if I had faster hardware I probably wouldn't notice the hit.



Initially, I am zoomed right out with the composite map covering the terrain. I then zoom into the terrain, and as you can see there is a huge drop in performance.

That is with parallax mapping disabled, no post effects, or anything else.

(Btw, do you know what that glitch in the terrain is? There's quite a few of those dotted around.)

[PaddyWagon]

That's why I develop on an NVidia GeForce 6600, because if I had faster hardware I probably wouldn't notice the hit.

awesome way to get things done and keep them quick, good call.

At the performance boundary your pixel shader takes a huge jump upwards, your driver time skyrockets and your pixel shader time goes up about 4x.

The most important thing here is that your batch count goes from almost nothing straight up to 2500+ batches, which means a lot of driver changes on each batch (explaining the driver time rise). What I think you're looking at is that you have way too small a size on the terrain blocks you're using and each one holds a material change.

If you could shade by material you'd probably see a 50x50 tiled quilt. A guess. In any case if that holds true then the best optimization would be to reduce batch count, either by making each tile of that quilt cover a much larger area or merging materials so fewer changes have to be made each frame. Also a guess.

GPU doesn't change at all, totally unstressed. If you were moving around during the capture I'd guess that the yellow framerate spikes are from data transfers to and from the video card when a pile of textures have to be cached out to main memory and more loaded in for the current scene.

The terrain glitch, that little cliff, may be precision loss or not, it's really hard to tell with the graphs covering it all and no indication of your world location and/or looking deeper into whether the source data has those glitches to begin with.

[DanielSefton]

Thanks for your perfhud summary

The most important thing here is that your batch count goes from almost nothing straight up to 2500+ batches, which means a lot of driver changes on each batch (explaining the driver time rise).

Yes, the RenderTarget::FrameStats batch figures I posted earlier were obviously wrong. 2500+ batches is worrying.

If you were moving around during the capture I'd guess that the yellow framerate spikes are from data transfers to and from the video card when a pile of textures have to be cached out to main memory and more loaded in for the current scene.

It was just a linear camera translation, zooming in from far away (where the initial rises are), then no movement at all. Though, I think that's a different issue altogether, because I remember having that every-few-seconds jump with MyRddin as well. Always wondered why it did that.

The terrain glitch, that little cliff, may be precision loss or not, it's really hard to tell with the graphs covering it all and no indication of your world location and/or looking deeper into whether the source data has those glitches to begin with.

Hopefully this is clearer; these little ridges are all over. The heightmap looks ok, but it's a PNG, can't be sure until I get a raw heightmap working.