[solved] Compositor is black with radeon 5700 xt

Problems building or running the engine, queries about how to use features etc.
Post Reply
User avatar
suny
Greenskin
Posts: 137
Joined: Thu Mar 12, 2020 5:53 pm
x 60

[solved] Compositor is black with radeon 5700 xt

Post by suny »

Ogre Version: :?: 1.12.5
Operating System: :?: windows10
Render System: :?: D9D

Hi,

I have several compositors in the SHMUP Creator, and they work with most PCs.
But with some gfx cards, the result of those compositors is black, and I don't know why.
For instance, with the radeon 5700 XT... There is no error message during the shaders compilation.

I'm doing this for the compositor:

Code: Select all

compositor pixelateFX
{
    technique
    {
        texture scene target_width target_height PF_BYTE_RGB
		
        // the scene we want to modulate
        target scene
        {
            input previous
        }



        target_output
        {
            input none
			shadows off

            pass render_quad
            {
                material modulatePixelate_out
                input 0 scene
            }
        }
    }
}
My material:

Code: Select all

vertex_program pixelate_out_vp hlsl
{
	source outline.hlsl
	entry_point outline_out_vp
    target vs_1_1 vs_2_0
    default_params
   	{
		param_named_auto worldViewProj worldviewproj_matrix
   	}
}

fragment_program pixelate_out hlsl
{
	source pixelate.hlsl
	entry_point pixelate_out
    target ps_3_0
    default_params
   	{
		 param_named pixelSize float 1
		 param_named_auto resoWidth viewport_width
		 param_named_auto resoHeigth viewport_height
   	}
}


material modulatePixelate_out
{
    technique
    {
        pass
        {
			vertex_program_ref pixelate_out_vp
			{
			}
			
			fragment_program_ref pixelate_out
			{
				param_named pixelSize float 1
				param_named_auto resoWidth viewport_width
				param_named_auto resoHeigth viewport_height
			}

            texture_unit
            {
				tex_address_mode border
                texture white.bmp
                filtering bilinear
				tex_address_mode clamp
            }
        }
    }
}
And my shader:

Code: Select all

struct VIn
{
    float4 p   : POSITION;
    float2 uv  : TEXCOORD0;
};

struct VOut
{
    float4 p   : SV_POSITION;
    float2 uv  : TEXCOORD0;
};

struct PIn
{
	float4 p   : SV_POSITION;
    float2 uv  : TEXCOORD0;
};

VOut pixelate_out_vs(VIn IN, uniform float4x4 wvp)
{
    VOut OUT;
    OUT.p = mul(wvp, IN.p);
    // clean up inaccuracies for the UV coords
    float2 uv = sign(IN.p);
    // convert to image space
    uv = (float2(uv.x, -uv.y) + 1.0) * 0.5;
    OUT.uv = uv;
    return OUT;
}


//-------------------------------
//Pixelate
//-------------------------------
float4 pixelate_out(float2 texCoord: TEXCOORD0,
		uniform float pixelSize,
		uniform float resoWidth ,
		uniform float resoHeigth ,
		uniform sampler RT: register(s0)
		) : COLOR {
	
	half dx = pixelSize*(1.0/resoWidth);
	half dy = pixelSize*(1.0/resoHeigth);
	texCoord = half2(dx*floor(texCoord.x/dx), dy*floor(texCoord.y/dy));
	
	float3 finalcol = tex2D(RT, texCoord).xyz;
		
	return float4(finalcol,1);
}
If anybody has an idea, I would be glad to know why!
S.
Last edited by suny on Thu Apr 22, 2021 7:59 am, edited 1 time in total.
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Compositor is black with radeon 5700 xt

Post by paroj »

- maybe PF_BYTE_RGB is not supported. Try PF_BYTE_RGBA.
- SV_POSITION is technically D3D10+. Try POSITION.
User avatar
suny
Greenskin
Posts: 137
Joined: Thu Mar 12, 2020 5:53 pm
x 60

Re: Compositor is black with radeon 5700 xt

Post by suny »

I tried your ideas, and still black with all my compositosr, on every AMD cards :/
What should I do to pinpoint what is wrong?

In my code I do this:

Code: Select all

postPIXELATE = Ogre::CompositorManager::getSingleton().addCompositor(playStatePtr->mviewport2, "pixelateFX");
postPIXELATE->setEnabled(level->IsPostPixel);
postPIXELATE->notifyResized();	
S.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5292
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Compositor is black with radeon 5700 xt

Post by dark_sylinc »

  1. Enable debug layer from DX9 panel. It may reveal the error
  2. Try changing OUT.p = mul(wvp, IN.p); for OUT.p = mul(wvp, float3(IN.p.xyz,1.0f));
  3. Get rid of the half, change it to float. RDNA cards actually support half, and that may be causing trouble
User avatar
suny
Greenskin
Posts: 137
Joined: Thu Mar 12, 2020 5:53 pm
x 60

Re: Compositor is black with radeon 5700 xt

Post by suny »

I tried 2 and 3 with no luck.

I newbie question: I don't know how to enable debug layer, and I can't find information about a DX9 panel.
How can I use this?
S.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5292
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Compositor is black with radeon 5700 xt

Post by dark_sylinc »

Mmmm, it's hard to find info now... I think it came with the Directx june 2010 sdk. I think it was named dxcpl.exe or dxpanel.cpl or something like that; somewhere in the installation folder (just look for .exe and .cpl files in the installed folder). If you're lucky it will install itself in the Control Panel.

It's also quite possible the debug runtimes will only work in Windows 7 (without KB 2370838) or XP. Update: Confirmed. It won't work in newer versions of Windows.

The panel looked like this.
paroj
OGRE Team Member
OGRE Team Member
Posts: 1993
Joined: Sun Mar 30, 2014 2:51 pm
x 1073
Contact:

Re: Compositor is black with radeon 5700 xt

Post by paroj »

yeah the best thing about D3D11 and GL3+ is that you get proper debug layers for them (e.g. renderdoc). If your app supports one of them, that might be worth giving a shot.

Otherwise you can always resort to poor-mans debugging: comment everything out until you get a trivial (working!) shader and start commenting-in things line-by-line.
User avatar
suny
Greenskin
Posts: 137
Joined: Thu Mar 12, 2020 5:53 pm
x 60

Re: Compositor is black with radeon 5700 xt

Post by suny »

I found a way to fix it:
I was using target ps_3_0 in the compositor's fragment shaders, and I changed it to target ps_2_b.

I'm happy, but I still don't know why those shaders were not working on AMD...
S.
User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 5292
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 1278
Contact:

Re: Compositor is black with radeon 5700 xt

Post by dark_sylinc »

Ohhh. Now I see it. You must pair ps 3.0 with vs 3.0
You were using VS 2.0

That's why it failed
Post Reply