[solved] Broken black shader on Ati

Problems building or running the engine, queries about how to use features etc.
Post Reply
Oogst
OGRE Expert User
OGRE Expert User
Posts: 1034
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
Contact:

[solved] Broken black shader on Ati

Post by Oogst » Mon Oct 26, 2009 10:50 pm

For Proun I have some post effects that require RenderTextures. This all works fine, except that on some Ati cards (or all? not sure) the post effect is rendered all black. Unfortunately, I don't have an Ati card myself, so I cannot quickly and extensively test this, so I am hoping you folks have some idea what is going wrong here. I ran this on several Nvidia cards, and so far all seem okay with these graphics, while the only two Ati cards I am aware of running this thing on, failed to do so.

The Ogre log does not give me any errors on the Ati cards. I don't know whether the issue is with the RenderTexture not being created/rendered, or with the shader not running correctly, since I don't get any errors. My overlays for lap time and such are rendered nicely, so rendering itself is working and the problem really seems to be with the post effect.

I hope these pieces of info give some insight into what might be going wrong here. This is how the RenderTexture is created:

Code: Select all

	Ogre::TexturePtr texture = Ogre::TextureManager::getSingleton().createManual(
		TextTools::append("RenderTexture", Global::getUniqueNumber()),
		Ogre::ResourceGroupManager::DEFAULT_RESOURCE_GROUP_NAME,
		Ogre::TEX_TYPE_2D, 
		width,
		height,
		0,
		Ogre::PF_R8G8B8A8,	
		Ogre::TU_RENDERTARGET,
		NULL);	//anti-aliasing samples
These are the Cg definitions of the shaders:

Code: Select all

vertex_program Cg_DoF_VP cg
{
	source DoF.cg
	entry_point DoF_VP
	profiles vs_2_0

	default_params
	{
		param_named_auto worldViewProjectionMatrix worldviewproj_matrix
	}
}

fragment_program Cg_DoF_SM3_32samples_FP cg
{
	source DoF.cg
	entry_point DoF_SM3_32samples_FP
	profiles ps_3_0

	default_params
	{
		param_named size			float 0.012
		param_named halfSize		float 0.006
		param_named hdriMultiplier	float 2.0
	}
}
And this is what the Ogre.log says about the Ati GPU:

Code: Select all

14:11:37: *-*-* OGRE Initialising
14:11:37: *-*-* Version 1.6.3 (Shoggoth)
14:11:37: Creating resource group Bootstrap
14:11:37: Added resource location 'Media/OgreCore' of type 'FileSystem' to resource group 'Bootstrap'
14:11:37: D3D9 : RenderSystem Option: Allow NVPerfHUD = No
14:11:37: D3D9 : RenderSystem Option: Anti aliasing = None
14:11:37: D3D9 : RenderSystem Option: Floating-point mode = Fastest
14:11:37: D3D9 : RenderSystem Option: Full Screen = No
14:11:37: D3D9 : RenderSystem Option: Rendering Device = ATI Radeon HD 4800 Series
14:11:37: D3D9 : RenderSystem Option: VSync = No
14:11:37: D3D9 : RenderSystem Option: Video Mode = 800 x 600 @ 32-bit colour
14:11:37: D3D9 : RenderSystem Option: sRGB Gamma Conversion = No
14:11:41: *** Stopping Win32GL Subsystem ***
14:11:41: CPU Identifier & Features
14:11:41: -------------------------
14:11:41:  *   CPU ID: GenuineIntel: Intel(R) Core(TM)2 Quad  CPU   Q9550  @ 2.83GHz
14:11:41:  *      SSE: yes
14:11:41:  *     SSE2: yes
14:11:41:  *     SSE3: yes
14:11:41:  *      MMX: yes
14:11:41:  *   MMXEXT: yes
14:11:41:  *    3DNOW: no
14:11:41:  * 3DNOWEXT: no
14:11:41:  *     CMOV: yes
14:11:41:  *      TSC: yes
14:11:41:  *      FPU: yes
14:11:41:  *      PRO: yes
14:11:41:  *       HT: no
14:11:41: -------------------------
14:11:41: D3D9 : Subsystem Initialising
14:11:41: D3D9RenderSystem::_createRenderWindow "OGRE Render Window", 800x600 windowed  miscParams: FSAA=0 FSAAQuality=0 colourDepth=32 gamma=false useNVPerfHUD=false vsync=false 
14:11:41: D3D9 : Created D3D9 Rendering Window 'OGRE Render Window' : 800x600, 32bpp
14:11:41: D3D9 : WARNING - disabling VSync in windowed mode can cause timing issues at lower frame rates, turn VSync on if you observe this problem.
14:11:41: Registering ResourceManager for type Texture
14:11:41: Registering ResourceManager for type GpuProgram
14:11:41: D3D9: Vertex texture format supported - PF_L8
14:11:41: D3D9: Vertex texture format supported - PF_L16
14:11:41: D3D9: Vertex texture format supported - PF_A8
14:11:41: D3D9: Vertex texture format supported - PF_A4L4
14:11:41: D3D9: Vertex texture format supported - PF_BYTE_LA
14:11:41: D3D9: Vertex texture format supported - PF_R5G6B5
14:11:41: D3D9: Vertex texture format supported - PF_B5G6R5
14:11:41: D3D9: Vertex texture format supported - PF_A4R4G4B4
14:11:41: D3D9: Vertex texture format supported - PF_A1R5G5B5
14:11:41: D3D9: Vertex texture format supported - PF_A8R8G8B8
14:11:41: D3D9: Vertex texture format supported - PF_B8G8R8A8
14:11:41: D3D9: Vertex texture format supported - PF_A2R10G10B10
14:11:41: D3D9: Vertex texture format supported - PF_A2B10G10R10
14:11:41: D3D9: Vertex texture format supported - PF_DXT1
14:11:41: D3D9: Vertex texture format supported - PF_DXT2
14:11:41: D3D9: Vertex texture format supported - PF_DXT3
14:11:41: D3D9: Vertex texture format supported - PF_DXT4
14:11:41: D3D9: Vertex texture format supported - PF_DXT5
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT16_RGB
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT16_RGBA
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT32_RGB
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT32_RGBA
14:11:41: D3D9: Vertex texture format supported - PF_X8R8G8B8
14:11:41: D3D9: Vertex texture format supported - PF_X8B8G8R8
14:11:41: D3D9: Vertex texture format supported - PF_R8G8B8A8
14:11:41: D3D9: Vertex texture format supported - PF_DEPTH
14:11:41: D3D9: Vertex texture format supported - PF_SHORT_RGBA
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT16_R
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT32_R
14:11:41: D3D9: Vertex texture format supported - PF_SHORT_GR
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT16_GR
14:11:41: D3D9: Vertex texture format supported - PF_FLOAT32_GR
14:11:41: D3D9: Vertex texture format supported - PF_SHORT_RGB
14:11:41: RenderSystem capabilities
14:11:41: -------------------------
14:11:41: RenderSystem Name: Direct3D9 Rendering Subsystem
14:11:41: GPU Vendor: ati
14:11:41: Device Name: ATI Radeon HD 4800 Series
14:11:41: Driver Version: 8.14.10.662
14:11:41:  * Fixed function pipeline: yes
14:11:41:  * Hardware generation of mipmaps: yes
14:11:41:  * Texture blending: yes
14:11:41:  * Anisotropic texture filtering: yes
14:11:41:  * Dot product texture operation: yes
14:11:41:  * Cube mapping: yes
14:11:41:  * Hardware stencil buffer: yes
14:11:41:    - Stencil depth: 8
14:11:41:    - Two sided stencil support: yes
14:11:41:    - Wrap stencil values: yes
14:11:41:  * Hardware vertex / index buffers: yes
14:11:41:  * Vertex programs: yes
14:11:41:  * Fragment programs: yes
14:11:41:  * Geometry programs: no
14:11:41:  * Supported Shader Profiles: hlsl ps_1_1 ps_1_2 ps_1_3 ps_1_4 ps_2_0 ps_2_a ps_2_b ps_2_x ps_3_0 vs_1_1 vs_2_0 vs_2_a vs_2_x vs_3_0
14:11:41:  * Texture Compression: yes
14:11:41:    - DXT: yes
14:11:41:    - VTC: no
14:11:41:  * Scissor Rectangle: yes
14:11:41:  * Hardware Occlusion Query: yes
14:11:41:  * User clip planes: yes
14:11:41:  * VET_UBYTE4 vertex element type: yes
14:11:41:  * Infinite far plane projection: yes
14:11:41:  * Hardware render-to-texture: yes
14:11:41:  * Floating point textures: yes
14:11:41:  * Non-power-of-two textures: yes
14:11:41:  * Volume textures: yes
14:11:41:  * Multiple Render Targets: 4
14:11:41:    - With different bit depths: yes
14:11:41:  * Point Sprites: yes
14:11:41:  * Extended point parameters: yes
14:11:41:  * Max Point Size: 256
14:11:41:  * Vertex texture fetch: yes
14:11:41:    - Max vertex textures: 4
14:11:41:    - Vertex textures shared: no
14:11:41:  * Render to Vertex Buffer : no
14:11:41:  * DirectX per stage constants: yes
14:11:41: ***************************************
14:11:41: *** D3D9 : Subsystem Initialised OK ***
14:11:41: ***************************************
Does anyone have any idea how to fix this, or how to find the problem? Thanks in advance! :)
Last edited by Oogst on Wed Oct 28, 2009 12:46 pm, edited 2 times in total.
0 x
blog.oogst3d.net: my dev blog and portfolio
Ronimo Games: my game dev company
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/PS3/PS4)
Swords & Soldiers: side-scrolling RTS (PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance

akem321
Bronze Sponsor
Bronze Sponsor
Posts: 153
Joined: Mon Jul 31, 2006 8:29 pm
Location: France
Contact:

Re: Black RenderTexture/shader on Ati

Post by akem321 » Tue Oct 27, 2009 2:58 am

Hi,
I believe there are chances it is a shader thingy, you could dump the RenderTexture to see if it's correctly renderered, i know Ogre has some easy way to do that like RenderTarget::writeContentsToTimestampedFile, things like that, may help...
I remember problems with my compositors not working on ATI because i was using unsupported functions, stuff that were working just fine on Nvidia; but it was glsl where you are using hlsl with DirectX here...
- ha it's Cg not hlsl...sry need to sleep :)
0 x

Oogst
OGRE Expert User
OGRE Expert User
Posts: 1034
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
Contact:

Re: Black RenderTexture/shader on Ati

Post by Oogst » Tue Oct 27, 2009 10:16 am

Ah, I didn't know about the writeContentsToFile function, so I am using that now to check things. I also asked the owner of the Ati to try my medium graphics setting, since that one support shader model 2.x, while the other settings are shader model 3.0. So I guess that way at least I'll find out whether the issue is with the RenderTexture or the shader.

Do you know of some site where I can see what 'standard' shader functionality Ati doesn't support? The Cg compiler doesn't complain, but these issues might come up at a later moment, simply by not rendering?
0 x
blog.oogst3d.net: my dev blog and portfolio
Ronimo Games: my game dev company
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/PS3/PS4)
Swords & Soldiers: side-scrolling RTS (PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance

Oogst
OGRE Expert User
OGRE Expert User
Posts: 1034
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
Contact:

Re: Black RenderTexture/shader on Ati

Post by Oogst » Tue Oct 27, 2009 2:06 pm

Okay, I got some further info. It turns out the RenderTexture renders okay, since the images it outputs with writeContentsToFile look good enough. :) So I think the problem must be with the shader. :(

Interestingly, it also doesn't work on a Material that has both a shader model 3.0 and a shader model 2.x implementation. I don't know which of the two the Ati card is trying to run, but the screen remains black. Since there is no error for the other materials (which do only shader model 3.0), I am guessing the Ati tries to run the shader model 3.0 implementation.

So, I think the great question then is: in what ways does Ati handle shaders differently?

I don't know if it helps, but this is the shader that is being run:

Code: Select all

void DoF_SM3_16samples_FP
	(
		float2 uv : TEXCOORD0,
		
		out float4 oColour : COLOR,

		uniform sampler2D	textureFullRes,
		uniform float		size,
		uniform float		halfSize,
		uniform float		hdriMultiplier
	)
{
	float2 taps[16] = {float2(0.130265, -0.26094),
						float2(0.323249, 0.269765),
						float2(-0.294219, 0.127005),
						float2(-0.364175, -0.258962),
						float2(0.412894, -0.145747),
						float2(0.0700567, 0.165608),
						float2(-0.118102, 0.411452),
						float2(-0.0503964, -0.17042),
						float2(0.159576, -0.0544302),
						float2(0.214983, 0.440148),
						float2(-0.330318, 0.350349),
						float2(-0.153879, -0.448555),
						float2(0.268464, -0.392533),
						float2(0.475753, 0.0517994),
						float2(-0.0817125, 0.0926575),
						float2(-0.365859, -0.0280665)};

	float lengths[16] = {0.291648,
						0.421026,
						0.320461,
						0.446862,
						0.437863,
						0.179817,
						0.428066,
						0.177716,
						0.168604,
						0.489845,
						0.481512,
						0.474216,
						0.475558,
						0.478564,
						0.123541,
						0.366934};

	//The central tap that defines the strength of the blur on this pixel
	float4 tapCentre = tex2D(textureFullRes, uv);

	//calculate the size of the blur on this pixel
	float sizeThisTap = size * tapCentre.a;

	oColour = tapCentre;

	//division is used to normalize the final colour, so it is the
	//addition of the weights for all taps.
	float division = 1;

	int i;
	for (i = 0; i < 16; ++i)
	{
		float4 tapColour = tex2D(textureFullRes, uv + sizeThisTap * taps[i]);
		float maximumAllowedDistance = tapColour.a * halfSize;
		
		if (lengths[i] * size <= maximumAllowedDistance)
		{
			division += 1;
			oColour += tapColour;
		}
	}

	oColour = oColour * hdriMultiplier / division;
}
Just wondering: could it be that each texture read should have its own sampler2D? Meaning that I should start with 17 times the line uniform sampler2D textureFullRes? Seems kind of crazy, but I am doing wild/random guessing here.
0 x
blog.oogst3d.net: my dev blog and portfolio
Ronimo Games: my game dev company
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/PS3/PS4)
Swords & Soldiers: side-scrolling RTS (PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance

nbeato
Gnome
Posts: 372
Joined: Thu Dec 20, 2007 1:00 am
Location: Florida
x 1
Contact:

Re: Broken black shader on Ati

Post by nbeato » Tue Oct 27, 2009 4:43 pm

Oogst wrote:Just wondering: could it be that each texture read should have its own sampler2D? Meaning that I should start with 17 times the line uniform sampler2D textureFullRes? Seems kind of crazy, but I am doing wild/random guessing here.
That is not the problem. I do that all the time and it works fine :) I would look at:

Code: Select all

uniform sampler2D   textureFullRes,
You never bind the texture. I don't know if that is even legal in CG? Try this:

Code: Select all

uniform sampler2D   textureFullRes : TEXUNIT0,
0 x

Oogst
OGRE Expert User
OGRE Expert User
Posts: 1034
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
Contact:

Re: Broken black shader on Ati

Post by Oogst » Tue Oct 27, 2009 5:51 pm

I have actually never seen that TEXUNIT0 semantic before, while I wrote quite a lot of shaders. Never heard about problems with the shaders in De Blob and my Interior Mapping demo. But this is the first time I am doing shader model 3.0, so maybe this has become a requirement in shader model 3.0? And maybe Nvidia doesn't care, while Ati does? I checked Nvidia's CG Manual document, and that does mention TEXUNIT0, but never uses it in the examples. That document is from 2005, though, so it doesn't discuss shader model 3.0 at all.

Anyway, tomorrow I have access to a couple of computers with Ati-cards, so then I can try whether this is indeed the problem. :) Does sound like a plausible cause for this problem, since the shader would still compile, but not be able to get actual colours from the texture.
0 x
blog.oogst3d.net: my dev blog and portfolio
Ronimo Games: my game dev company
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/PS3/PS4)
Swords & Soldiers: side-scrolling RTS (PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance

nbeato
Gnome
Posts: 372
Joined: Thu Dec 20, 2007 1:00 am
Location: Florida
x 1
Contact:

Re: Broken black shader on Ati

Post by nbeato » Tue Oct 27, 2009 6:58 pm

Well, I can say about 2-3 years ago when I switched from GLSL to CG, I had problems with shaders using multiple textures (using nvidia cards) because I did not bind the texture in the CG file. You can use TEXUNIT* or register(s*). Either way, hopefully it fixes your problem. Keep us posted.
0 x

Oogst
OGRE Expert User
OGRE Expert User
Posts: 1034
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
Contact:

Re: Broken black shader on Ati

Post by Oogst » Wed Oct 28, 2009 9:10 am

I had a chance to work with an Ati today and the conclusion is that the TEXUNIT0 thing is not the problem. While experimenting, I went as far as to strip down the shader to this really simple thing:

Code: Select all

void DoF_SM3_16samples_FP(out float4 oColour : COLOR, uniform float hdriMultiplier)
{
	oColour = float4(1.0f, hdriMultiplier, 0.0f, 0.0f);
}
When I have Ogre compile this as a shader 3.0, it still gives a black screen. When I let it compile as shader 2.x, it gives yellow, which is correct. On Nvidia cards, it always gives yellow, so always correct. My depth of field implementation for shader model 2.x also works fine on Ati cards, it is just the (much better) 3.0 implementations that don't work.

I also added TEXUNITx to all the other shaders and fixed all the warnings (float4 to float3, mainly), and that did not help either. I then changed all the 0's to 0.0f and such where floats were meant (read somewhere that that was important for Ati's). Didn't help either.

So, I am kind of out of options here. I cannot go any simpler than that shader and Ati's still only work with shader model 2.x, not with 3.0.

By the way, by now I have the same problem reported on six different Ati cards and not on any Nvidia cards, and no one with an Ati card claimed it works. So the problem really is specifically with Ati cards.

Would anyone have any idea what to do with this?
0 x
blog.oogst3d.net: my dev blog and portfolio
Ronimo Games: my game dev company
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/PS3/PS4)
Swords & Soldiers: side-scrolling RTS (PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance

Vectrex
Ogre Magi
Posts: 1266
Joined: Tue Aug 12, 2003 1:53 am
Location: Melbourne, Australia
Contact:

Re: Broken black shader on Ati

Post by Vectrex » Wed Oct 28, 2009 10:03 am

You'll kick yourself if this works, like I did.
You can't have different vs and ps profiles on ati cards :) You have v2.0 vs and v3.0 ps.
I really think OGRE should warn in the log about this if it finds it.
0 x

Oogst
OGRE Expert User
OGRE Expert User
Posts: 1034
Joined: Mon Mar 29, 2004 8:49 pm
Location: the Netherlands
Contact:

Re: Broken black shader on Ati

Post by Oogst » Wed Oct 28, 2009 12:43 pm

YES! That did it, thanks! I don't see why I should kick myself, though: not something I should have known, as far as I know. And definitely something that Ogre should put a warning for, I think. This thing is too deep in the engine for me to create a patch, so I will post a bug report for this.
Last edited by Oogst on Wed Oct 28, 2009 1:54 pm, edited 1 time in total.
0 x
blog.oogst3d.net: my dev blog and portfolio
Ronimo Games: my game dev company
Awesomenauts: platforming MOBA (PC/Mac/Linux/XBox360/PS3/PS4)
Swords & Soldiers: side-scrolling RTS (PS3/Wii/PC/Mac/Linux/iPhone/iPad/Android)
Proun: abstract racing game (PC)
Cello Fortress: mixing game and live cello performance

Vectrex
Ogre Magi
Posts: 1266
Joined: Tue Aug 12, 2003 1:53 am
Location: Melbourne, Australia
Contact:

Re: [solved] Broken black shader on Ati

Post by Vectrex » Wed Oct 28, 2009 1:21 pm

hehe great. That took us 3 days on a deadline to figure that out.
0 x

User avatar
mkultra333
Gold Sponsor
Gold Sponsor
Posts: 1889
Joined: Sun Mar 08, 2009 5:25 am
x 36

Re: Broken black shader on Ati

Post by mkultra333 » Wed Oct 28, 2009 3:07 pm

Vectrex wrote:You'll kick yourself if this works, like I did.
You can't have different vs and ps profiles on ati cards :) You have v2.0 vs and v3.0 ps.
I really think OGRE should warn in the log about this if it finds it.
Whoa! That's one nasty "gotcha." Glad I read that, I think my profiles are all over the place but since I'm using nvidia I never had a problem. I better do some fixing...
0 x
"In theory there is no difference between practice and theory. In practice, there is." - Psychology Textbook.

nbeato
Gnome
Posts: 372
Joined: Thu Dec 20, 2007 1:00 am
Location: Florida
x 1
Contact:

Re: [solved] Broken black shader on Ati

Post by nbeato » Thu Oct 29, 2009 10:57 pm

That's awesome it is fixed... but wow... talk about crazy. Good job Vectrex, you probably will save a few people headaches with that response!
0 x

Major Clod
Gnoblar
Posts: 9
Joined: Tue Mar 04, 2008 11:33 am

Re: [solved] Broken black shader on Ati

Post by Major Clod » Thu Nov 12, 2009 12:53 am

I've just had this same problem, I definately wouldn't have found this easily without this thread. Thanks!
0 x

User avatar
sinbad
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 19265
Joined: Sun Oct 06, 2002 11:19 pm
Location: Guernsey, Channel Islands
x 2
Contact:

Re: [solved] Broken black shader on Ati

Post by sinbad » Wed Dec 23, 2009 3:31 pm

Guys, I have 3 words for you: DirectX Debug Runtime.

Everyone who is deploying on DirectX should be running their apps through the debug runtime on a regular basis. This would have hilighted the problem before you got anywhere near the ATI machine, and will also alert you to any number of other problems. It's a PITA for us to try to detect all the edge-cases that you can possibly have on different hardware, and just squirreling something away in the log probably wouldn't do any good anyway, because people don't read the log until they have a problem. Running your app via the DX Debug Runtime shoves problems like this in your face early and makes you solve them. Consider this issue to be a lesson that you should be using this free tool regularly if you're serious about developing under DirectX.

And for the record, I've hit this problem myself before too, in fact only recently with the terrain work when I added a SM3 pixel shader path without changing the vertex shaders (still SM2). The Dx9 Debug Runtime alerted me to my mistake before I tested on ATI and was far more effective than burying a warning in the log.
0 x

Post Reply