reading back the stencil buffer in Ogre 2.1

Discussion area about developing with Ogre2 branches (2.1, 2.2 and beyond)
Post Reply
User avatar
Daixiwen
Greenskin
Posts: 104
Joined: Fri Feb 08, 2013 11:30 am
Location: Oslo
x 14

reading back the stencil buffer in Ogre 2.1

Post by Daixiwen »

Hi!

I'm trying to use stencil compositor passes in Ogre 2.1 (using OpenGL on Windows) but I can't get it to work exactly as I would like. I'm probably missing something obvious but I can't figure it out.
I'd like to make a postprocessing effect on some selected items. For that I created a new render queue and moved the select items to that new queue. Then I added stencil passes to modify the stencil buffer only when those items are drawn and a postprocessing render_quad pass that would only render on the parts where the stencil buffer has been modified. So far so good. Here is my compositor script:

Code: Select all

compositor_node MainDisplayRenderingNode
{
	in 0 rt_renderwindow
	target rt_renderwindow
	{
		pass clear
		{
			colour_value 0.2 0.2 0.9 1
			stencil_value 0
		}

		//Render opaque stuff
		pass render_scene
		{
			overlays	off
			rq_first	0
			rq_last		2
			shadows		PbsMaterialsShadowNode
		}

		// first highlight pass
		pass stencil
		{
			check		on
			comp_func	always_pass
			ref_value	0
			fail_op		keep
			depth_fail_op keep
			pass_op		invert
			two_sided off
		}

		pass render_scene
		{
			overlays	off
			rq_first	2
			rq_last		3
			shadows		PbsMaterialsShadowNode
		}

		pass stencil
		{
			check		on
			comp_func	not_equal
			ref_value	0
			fail_op		keep
			depth_fail_op keep
			pass_op		keep
			two_sided off
		}

		pass render_quad
		{
			material Highlight
		}

		pass stencil
		{
			check		off
		}

		// Render sky after opaque stuff (performance optimization)
		pass render_quad
		{
			quad_normals	camera_direction
			material SkyPostprocess
		}

		pass custom MYGUI
		{
		}
	}

	out 0 rt_renderwindow
}

compositor_node_shadow MainDisplayShadowNode
{
	technique pssm

	num_splits		3
	pssm_lambda		0.95
	shadow_map 0 2048 2048 PF_D32_FLOAT light 0 split 0
	shadow_map 1 1024 1024 PF_D32_FLOAT light 0 split 1
	shadow_map 2 1024 1024 PF_D32_FLOAT light 0 split 2

	technique focused
	shadow_map 4 2048 2048 PF_D32_FLOAT light 1
	shadow_map 5 2048 2048 PF_D32_FLOAT light 2

	shadow_map 0 1 2 4 5
	{
		pass clear
		{
			colour_value 1 1 1 1
		}
		pass render_scene
		{
		}
	}
}

workspace MainDisplayWorkspace
{
	connect_output MainDisplayRenderingNode 0
}
It works like a charm. Now I'd like a nicer effect, creating a halo around the highlighted items. I saw some examples on the net where they render a second version of the item to highlight with a slightly bigger dimension and use that as a stencil buffer. I don't really like that solution, it seems a bit overkill to render the items a second time, and besides it doesn't look nice on non trivial shapes.
Then I saw the tutorial "ReconstructPosFromDepth" and I thought 'hey! why not simply read the stencil buffer from my postprocessing effect!'. Then I can do whatever I want, like applying convolution filters and do everything in 2d. So I adapted the tutorial to my case. Now I can read the depth buffer without any trouble, but I can't read the stencil buffer. While debugging, I came to realize that none of the stencil pass parameters has any effect. I ended up trying this:

Code: Select all

compositor_node MainDisplayRenderingNode
{
	in 0 rt_renderwindow
	texture rt0 target_width target_height PF_R8G8B8 depth_texture depth_format PF_D24_UNORM_S8_UINT depth_pool 2
	texture depthTexture target_width target_height PF_D24_UNORM_S8_UINT depth_pool 2
	texture depthTextureCopy target_width target_height PF_D24_UNORM_S8_UINT

	target rt0
	{		
		pass clear
		{
			colour_value 0.2 0.2 0.9 1
			stencil_value 0
		}

		//Render opaque stuff
		pass render_scene
		{
			overlays	off
			rq_first	0
			rq_last		2
			shadows		PbsMaterialsShadowNode
		}

		// first highlight pass
		pass stencil
		{
			check		on
			comp_func	always_fail
			ref_value	0
			fail_op		keep
			depth_fail_op keep
			pass_op		invert
			two_sided off
		}

		pass render_scene
		{
			overlays	off
			rq_first	2
			rq_last		3
			shadows		PbsMaterialsShadowNode
		}

		pass stencil
		{
			check		off
		}

		// Render sky after opaque stuff (performance optimization)
		pass render_quad
		{
			quad_normals	camera_direction
			material SkyPostprocess
		}
	}

	target rt_renderwindow
	{
		pass depth_copy
		{
			alias_on_copy_failure	on
			in		depthTexture
			out		depthTextureCopy
		}

		pass render_quad
		{
			material Highlight
			input 0 rt0
			input 1 depthTextureCopy
			quad_normals camera_far_corners_world_space_centered
		}

		pass custom MYGUI
		{
		}
	}

	out 0 rt_renderwindow
}

compositor_node_shadow MainDisplayShadowNode
{
	technique pssm

	num_splits		3
	pssm_lambda		0.95
	shadow_map 0 2048 2048 PF_D32_FLOAT light 0 split 0
	shadow_map 1 1024 1024 PF_D32_FLOAT light 0 split 1
	shadow_map 2 1024 1024 PF_D32_FLOAT light 0 split 2

	technique focused
	shadow_map 4 2048 2048 PF_D32_FLOAT light 1
	shadow_map 5 2048 2048 PF_D32_FLOAT light 2

	shadow_map 0 1 2 4 5
	{
		pass clear
		{
			colour_value 1 1 1 1
		}
		pass render_scene
		{
		}
	}
}

workspace MainDisplayWorkspace
{
	connect_output MainDisplayRenderingNode 0
}
Notice the "always_fail" in the first stencil pass. This should prevent anything from render queue 2 to appear, but it still does. I haven't found any message in the Ogre log related to the depth or stencil buffer. Doing the same thing in the first version of the compositor script hides the items as expected.
Does anyone has an idea on what I'm doing wrong? Could it be that the depth texture that I create doesn't have a stencil component? In that case how can I create it with stencil?

Thanks a lot!
Hardware, n.: part of the computer you can kick

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4211
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 802
Contact:

Re: reading back the stencil buffer in Ogre 2.1

Post by dark_sylinc »

Hi!

A few things:

Stencil buffer in 2.1 is among the least tested features, so it could be not your fault.
Regarding the stencil not working in the 2nd compo, I don't know what to say. RenderDoc may help a lot in diagnosing the contents & state of the stencil buffer (and for starters... if it exists!). Also checking in D3D11 may provide more useful info in a debug build due to its thorough validation layers.

As for reading the stencil component as a texture in the shaders: AFAIK it's not possible in D3D11 (I'll be happy to be proven otherwise) and I've read that in GL it's possible but available documentation & examples are scarce, so getting it to work properly would be a nightmare. Therefore in Ogre it's possible to read the depth contents by binding the depth texture to the shader, but not the stencil part.

User avatar
Daixiwen
Greenskin
Posts: 104
Joined: Fri Feb 08, 2013 11:30 am
Location: Oslo
x 14

Re: reading back the stencil buffer in Ogre 2.1

Post by Daixiwen »

Hehe that what happens when you put so many awesome features in Ogre 2.1, now we want more :D. I read somewhere (but can't find the link again) that when using a 24/8 depth buffer, we could read back in the shader the depth value in x and the stencil in w. But it looks like it's more complicated than that. Well I'll look for another way to make my effect. Thanks for the explanation!

I'll try running with D3D11 and see what I get. I just have a shader to convert to hlsl and I also have a problem with the cubemapped sky (will probably come in another thread once I managed to debug it a bit more).
I had installed RenderDoc before begining this thread but couldn't get it to work. After investigating a bit it seems I have to disable the OpenGL plugin or RenderDoc will hang the process at startup. It does it too with the Ogre samples so it doesn't come from my app. The log doesn't help a lot

Code: Select all

22:49:21: Loading library .\RenderSystem_Direct3D11
22:49:21: Installing plugin: D3D11 RenderSystem
22:49:21: D3D11 : Direct3D11 Rendering Subsystem created.
22:49:21: D3D11: Driver Detection Starts
22:49:21: D3D11: Driver Detection Ends
22:49:21: Plugin successfully installed
22:49:21: Loading library .\RenderSystem_GL3Plus
22:49:21: Installing plugin: GL 3+ RenderSystem
22:49:21: OpenGL 3+ Rendering Subsystem created.
I'll try and run it in D3D11 and say what I find out with RenderDoc.
Hardware, n.: part of the computer you can kick

User avatar
Daixiwen
Greenskin
Posts: 104
Joined: Fri Feb 08, 2013 11:30 am
Location: Oslo
x 14

Re: reading back the stencil buffer in Ogre 2.1

Post by Daixiwen »

When running in D3D11, the stencil pass does work in both compo settings, so it looks like I only get this problem with OpenGL.
Is there any setting or known workaround that is needed to make RenderDoc work with the OpenGL Ogre plugin?
Hardware, n.: part of the computer you can kick

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4211
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 802
Contact:

Re: reading back the stencil buffer in Ogre 2.1

Post by dark_sylinc »

Daixiwen wrote:When running in D3D11, the stencil pass does work in both compo settings, so it looks like I only get this problem with OpenGL.
Ahh, likely an Ogre bug then (my bet no stencil buffer is being created...).
Will check in a some time later as right now I'm on crunch mode.
Daixiwen wrote:Is there any setting or known workaround that is needed to make RenderDoc work with the OpenGL Ogre plugin?
Works fine for me with my programs and the main samples. I'm on an AMD machine.
Try checking RenderDoc's log (Window->Log Errors and Warnings).

If not, contact baldur directly sending your sample. First try to see if you can get it to run on another machine. He's often very open and gets very curious with apps that don't work with his tool. We've been working close together to make RenderDoc very compatible with Ogre, while reporting many RenderDoc bugs.

User avatar
dark_sylinc
OGRE Team Member
OGRE Team Member
Posts: 4211
Joined: Sat Jul 21, 2007 4:55 pm
Location: Buenos Aires, Argentina
x 802
Contact:

Re: reading back the stencil buffer in Ogre 2.1

Post by dark_sylinc »

I took a look.

There are two things that enter into play here:

1. There was an Ogre bug in RenderSystems/GL3Plus/src/OgreGL3PlusDepthBuffer.cpp:

Code: Select all

if( mStencilBufferName )
{
	OCGE( glFramebufferRenderbuffer( target, GL_STENCIL_ATTACHMENT,
									 GL_RENDERBUFFER, mDepthBufferName ) );
}
Should be:

Code: Select all

if( mStencilBufferName )
{
	OCGE( glFramebufferRenderbuffer( target, GL_STENCIL_ATTACHMENT,
									 GL_RENDERBUFFER, mStencilBufferName ) );
}
This bug was fixed in branch 2.1-pso which will soon be merged with the main 2.1.
If you don't want to wait, either checkout that branch or port that single line change :)

2. OpenGL doesn't allow sharing the depth buffer with the main render target. This is explained in the comments of RSC_RTT_MAIN_DEPTHBUFFER_ATTACHABLE. Basically, with a few exceptions, you will have to use an intermediary RTT so that you can share the texture.

There is a new sample "StencilTest" in the 2.1-pso branch; you may want to check it out (note: the PSO branch isn't really needed, the code should work in the regular branch too).

Cheers

User avatar
Daixiwen
Greenskin
Posts: 104
Joined: Fri Feb 08, 2013 11:30 am
Location: Oslo
x 14

Re: reading back the stencil buffer in Ogre 2.1

Post by Daixiwen »

Thanks, I wasn't expecting you to look into this so quickly!
I started to make a stencil sample to help you debug the problem but it looks like you don't need it any more.
Hardware, n.: part of the computer you can kick

User avatar
Daixiwen
Greenskin
Posts: 104
Joined: Fri Feb 08, 2013 11:30 am
Location: Oslo
x 14

Re: reading back the stencil buffer in Ogre 2.1

Post by Daixiwen »

I saw what you did in the stencil test and I think I should be able to use something similar to do the effect I wanted.
I'' wait until the pso branch is merged into 2.1 before testing it again through. I tried your modification in 1. but with opengl I still can only use the stencil buffer on the main render target but not on any other render target. My Ogre branch is a few weeks old though and it's possible there are other parts of the pso branch that are required. I'll keep you updated when I test this again!
Hardware, n.: part of the computer you can kick

Post Reply