Silhouette Edge Highlighting

Problems building or running the engine, queries about how to use features etc.
Post Reply
gkimsey
Kobold
Posts: 34
Joined: Wed Feb 02, 2011 11:10 pm
x 1

Silhouette Edge Highlighting

Post by gkimsey »

Hi all. I'm checking the feasibility of using edge highlighting in my game. I've used a lot of wiki example code and ideas from previous forum posts to create the following function.
A few lines of this are specific to my implementation (ones that use unitInTurn), but that should be obvious.

Code: Select all

{
	size_t vertex_count,index_count;
	int edgeNum = 0;
	Ogre::Vector3* vertices;
    unsigned long * indices;

	// This function is directly from an Ogre Wiki entry and is just used to get the vertex list.
    getMeshInformation(unitInTurn->entity->getMesh(),vertex_count,vertices,index_count,indices);

    Ogre::Vector3 *vertexArray = new Ogre::Vector3[vertex_count];
    vertexArray = vertices;

	// Setup for edge detection
	unitInTurn->entity->getMesh()->buildEdgeList();
	Ogre::EdgeData * ed = unitInTurn->entity->getMesh()->getEdgeList();
	Ogre::Vector4 cameraLight;
	cameraLight.x = app->mCamera->getPosition().x;
	cameraLight.y = app->mCamera->getPosition().y;
	cameraLight.z = app->mCamera->getPosition().z;
	cameraLight.w = 1.0;
	ed->updateTriangleLightFacing(Ogre::Vector4(cameraLight));

	// Setup for line creation
	Ogre::SceneNode* myManualObjectNode = unitInTurn->node->createChildSceneNode("edge_node"); 
	Ogre::MaterialPtr myManualObjectMaterial = Ogre::MaterialManager::getSingleton().create("manual1Material","General"); 
	myManualObjectMaterial->setReceiveShadows(false); 
	myManualObjectMaterial->getTechnique(0)->setLightingEnabled(true); 
	myManualObjectMaterial->getTechnique(0)->getPass(0)->setDiffuse(0,0,1,0); 
	myManualObjectMaterial->getTechnique(0)->getPass(0)->setAmbient(0,0,1); 
	myManualObjectMaterial->getTechnique(0)->getPass(0)->setSelfIllumination(0,0,1);

	// For all edges in this entity...
	for (int i = 0; i < (int)ed->edgeGroups.size(); i++)
	{
		Ogre::EdgeData::EdgeGroup * group = &ed->edgeGroups[i];
		for (int j = 0; j < (int)group->edges.size(); j++)
		{
			if (!group->edges[j].degenerate && (ed->triangleLightFacings[group->edges[j].triIndex[0]] != ed->triangleLightFacings[group->edges[j].triIndex[1]]))
			{
				// Edge should be highlighted.
				Ogre::ManualObject* myManualObject =  app->mSceneMgr->createManualObject(); 

				myManualObject->begin("manual1Material", Ogre::RenderOperation::OT_LINE_LIST); 
				Ogre::Vector3 lineVertex1 = vertexArray[group->edges[j].vertIndex[0]];
				Ogre::Vector3 lineVertex2 = vertexArray[group->edges[j].vertIndex[1]];
				myManualObject->position(lineVertex1);
				myManualObject->position(lineVertex2);
				myManualObject->end(); 
 
				myManualObjectNode->attachObject(myManualObject);

				edgeNum++;
			}
		}
	}
}
Obviously 90% of this is code I ripped off from somewhere else. I'm just trying to get an idea of whether this a) is possible and b) will give me the result I want.

The code works, but with a few issues:
1) Each sub-entity gets its own edges highlighted. I knew this would happen, but it certainly ruins the effect and I'm not sure how to "fix" it so that the whole entity's silhouette is highlighted instead of the individual parts.
2) Line thickness is an issue. I think my mesh may be partly occluding the lines, so sometimes I don't even get the full 1-pixel-wide line.

I also haven't tested the efficiency of this function (I'd have to pull out the buildEdgeList() part since that should only have to be done once).

In digging around the forums I realized that what I want is what Assaf Raman posted here (first post on the second page, ogre head with white border):
http://www.ogre3d.org/forums/viewtopic. ... 1&start=25
Unfortunately he didn't post source to that particular example. Whatever he did eliminates my problem #2, although I'm guessing he's using OpenGL's line width parameter, and I'm not sure I want to be tied to a particular renderer yet.

I also have to say I'm surprised there's not a wiki entry on this already (there is a talk page about a similar goal of highlighting the borders of a 2D map. This was part of the code I stole). I'd be glad to make one myself if I can get this thing working.

Any help? Suggestions? Similar issues?

User avatar
Shockeye
Gremlin
Posts: 154
Joined: Mon Nov 24, 2008 10:34 am
Location: 'Straya
x 1

Re: Silhouette Edge Highlighting

Post by Shockeye »

Assaf posted code here using stencil outlines.

Or using edgelists, maybe you could do some processing to remove common edges from submeshes, merging them into one

gkimsey
Kobold
Posts: 34
Joined: Wed Feb 02, 2011 11:10 pm
x 1

Re: Silhouette Edge Highlighting

Post by gkimsey »

Shockeye wrote:Assaf posted code here using stencil outlines.

Or using edgelists, maybe you could do some processing to remove common edges from submeshes, merging them into one
Thanks, but unfortunately this is a different process that won't give me what I need. The "glow" effect there only really works for meshes with something approaching radial symmetry. Imagine applying that effect to the letter "F" in 2D, for instance. The glow at the top horizontal bar of the F would be way above it, the glow at the middle horizontal bar would be almost invisible, and the glow at the vertical bar would be to the left. That's just an example, but suffice to say the meshes I need to highlight are not happily round or semi-radial like the ogrehead.mesh.

Also, my submeshes don't necessarily share edges with each other that I could eliminate. I think in some cases that's probably true, but not always. It's unfortunate because that would actually be a pretty easy thing to look for.

It has occurred to me that the problem of "finding lines bounded completely by other lines" has probably been addressed before, and if there were a relatively efficient way to do it that would probably work. I can imagine outlier cases where there is an edge that is a valid outline for part of its length but not valid for another part, but it's difficult to predict how noticeable or common that would be.

It's a bit frustrating knowing that people have solved this problem dozens of times, even 10+ years ago, and I just don't know how they're doing it.

I did come up with one possible solution for the submesh outlines, which is to do something akin to a ray query for each vertex, and not highlight edges where a ray to one of its vertexes intersects the model. This would be pretty expensive just doing it against the bounding box, but it's probably tons more expensive doing it for each triangle (which I'd have to -- there's no way the bounding box thing wouldn't throw false positives).

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: Silhouette Edge Highlighting

Post by Assaf Raman »

Try having a look at this RenderMonkey sample:
"C:\Program Files (x86)\AMD\RenderMonkey 1.82\Examples\Dx9\NPR.rfx" - sub sample - DilateErode
Watch out for my OGRE related tweets here.

gkimsey
Kobold
Posts: 34
Joined: Wed Feb 02, 2011 11:10 pm
x 1

Re: Silhouette Edge Highlighting

Post by gkimsey »

Assaf Raman wrote:Try having a look at this RenderMonkey sample:
"C:\Program Files (x86)\AMD\RenderMonkey 1.82\Examples\Dx9\NPR.rfx" - sub sample - DilateErode
Holy crap, that's almost exactly what I need!
... but even assuming I'm licensed to use that, I have no idea how to do so. D: I'm looking into it, but if anyone can help me go from exactly zero knowledge of shaders to exporting and using this RenderMonkey example in my project, I would be very appreciative.

EDIT:
I discovered RmOgreExporter, but no built version. Tried to build it from source but failed miserably. There are references to python all over the project but the wiki and readme are explicit that all I need are Boost, Loki, and the RenderMonkey SDK. I installed Python anyway which got me past one error but now I get:
1>..\..\src\PyRmGatewayPlugin\PyRmGatewayEntry.cpp(42): error C3861: 'OutputDebugStr': identifier not found

Does anyone have a built version of the exporter plugin? I get the feeling I could spend a full week just getting the exporter to build only to find out it doesn't work for the sample I want anyway.

User avatar
Assaf Raman
OGRE Team Member
OGRE Team Member
Posts: 3092
Joined: Tue Apr 11, 2006 3:58 pm
Location: TLV, Israel
x 76

Re: Silhouette Edge Highlighting

Post by Assaf Raman »

In this case you can't just "export" from render monkey - so don't waste your time on the exporter.
What you need to do is create an OGRE compositor based on the shaders in that sample.
The shaders there are very basic - so need for a license.
Try to learn the sample - then try to port it to OGRE - shouldn't be that hard.
Watch out for my OGRE related tweets here.

gkimsey
Kobold
Posts: 34
Joined: Wed Feb 02, 2011 11:10 pm
x 1

Re: Silhouette Edge Highlighting

Post by gkimsey »

I gotcha. Thanks a lot for your help. I could've wasted a whole lot of time on that build. =D
I'll see if I can port it over as you suggested. Certainly if it works I'll be sure to add it to this post and the wiki.

gkimsey
Kobold
Posts: 34
Joined: Wed Feb 02, 2011 11:10 pm
x 1

Re: Silhouette Edge Highlighting

Post by gkimsey »

I'm feeling pretty stuck here. When I dug into this some more and read up on compositors and materials, I couldn't see any reason why I should be porting this to a compositor. I want to highlight a single model at a time, not apply a full-screen effect.

So I tried porting the DilateErode to a .material script. Below is the exported .fx

Code: Select all

//--------------------------------------------------------------//
// DilateErode
//--------------------------------------------------------------//
//--------------------------------------------------------------//
// DrawObject
//--------------------------------------------------------------//
string NPR_DilateErode_DrawObject_Elephant : ModelData = "..\\..\\..\\..\\Program Files (x86)\\AMD\\RenderMonkey 1.82\\Examples\\Media\\Models\\ElephantBody.3ds";

texture ImageRT_Tex : RenderColorTarget
<
   float2 ViewportRatio={1.0,1.0};
   string Format="D3DFMT_A8R8G8B8";
   float  ClearDepth=1.000000;
   int    ClearColor=0;
>;
float4x4 view_proj_matrix : ViewProjection;
struct VS_OUTPUT {
   float4 Pos: POSITION;
};

VS_OUTPUT NPR_DilateErode_DrawObject_Vertex_Shader_main(float4 Pos: POSITION){
   VS_OUTPUT Out;

   // Standard mvp transform
   Out.Pos = mul(view_proj_matrix, Pos);

   return Out;
}



float4 NPR_DilateErode_DrawObject_Pixel_Shader_main() : COLOR {
   // Output white   
   return 1;
}


//--------------------------------------------------------------//
// Dilate
//--------------------------------------------------------------//
string NPR_DilateErode_Dilate_ScreenAlignedQuad : ModelData = "..\\..\\..\\..\\Program Files (x86)\\AMD\\RenderMonkey 1.82\\Examples\\Media\\Models\\ScreenAlignedQuad.3ds";

float pixelSize
<
   string UIName = "pixelSize";
   string UIWidget = "Numeric";
   bool UIVisible =  true;
   float UIMin = 0.00;
   float UIMax = 0.01;
> = float( 0.00 );
struct NPR_DilateErode_Dilate_Vertex_Shader_VS_OUTPUT {
   float4 Pos: POSITION;
   float2 texCoord: TEXCOORD;
};

NPR_DilateErode_Dilate_Vertex_Shader_VS_OUTPUT NPR_DilateErode_Dilate_Vertex_Shader_main(float4 Pos: POSITION){
   NPR_DilateErode_Dilate_Vertex_Shader_VS_OUTPUT Out;

   // Clean up inaccuracies
   Pos.xy = sign(Pos.xy);

   Out.Pos = Pos;
   Out.texCoord.x = 0.5 * (1 + Pos.x + pixelSize);
   Out.texCoord.y = 0.5 * (1 - Pos.y + pixelSize);

   return Out;
}




float NPR_DilateErode_Dilate_Pixel_Shader_pixelSize
<
   string UIName = "NPR_DilateErode_Dilate_Pixel_Shader_pixelSize";
   string UIWidget = "Numeric";
   bool UIVisible =  true;
   float UIMin = 0.00;
   float UIMax = 0.01;
> = float( 0.00 );
sampler ImageRT = sampler_state
{
   Texture = (ImageRT_Tex);
   ADDRESSU = CLAMP;
   ADDRESSV = CLAMP;
   MAGFILTER = LINEAR;
   MINFILTER = LINEAR;
   MIPFILTER = LINEAR;
};
// The surrounding pixels
float2 samples[8] = {
   -1, -1,
    0, -1,
    1, -1,
   -1,  0,
    1,  0,
   -1,  1,
    0,  1,
    1,  1,
};

float4 NPR_DilateErode_Dilate_Pixel_Shader_main(float2 texCoord: TEXCOORD) : COLOR {
   // Simple dilate operation, find the maximum
   // pixel in the neighborhood.
   float4 maxSamp = tex2D(ImageRT, texCoord);
   for (int i = 0; i < 8; i++){
      float4 sample = tex2D(ImageRT, texCoord + NPR_DilateErode_Dilate_Pixel_Shader_pixelSize * samples[i]);
      maxSamp = max(maxSamp, sample);
   }

   return maxSamp;
}







//--------------------------------------------------------------//
// Erode
//--------------------------------------------------------------//
string NPR_DilateErode_Erode_ScreenAlignedQuad : ModelData = "..\\..\\..\\..\\Program Files (x86)\\AMD\\RenderMonkey 1.82\\Examples\\Media\\Models\\ScreenAlignedQuad.3ds";

float NPR_DilateErode_Erode_Vertex_Shader_pixelSize
<
   string UIName = "NPR_DilateErode_Erode_Vertex_Shader_pixelSize";
   string UIWidget = "Numeric";
   bool UIVisible =  true;
   float UIMin = 0.00;
   float UIMax = 0.01;
> = float( 0.00 );
struct NPR_DilateErode_Erode_Vertex_Shader_VS_OUTPUT {
   float4 Pos: POSITION;
   float2 texCoord: TEXCOORD;
};

NPR_DilateErode_Erode_Vertex_Shader_VS_OUTPUT NPR_DilateErode_Erode_Vertex_Shader_main(float4 Pos: POSITION){
   NPR_DilateErode_Erode_Vertex_Shader_VS_OUTPUT Out;

   // Clean up inaccuracies
   Pos.xy = sign(Pos.xy);

   Out.Pos = Pos;
   Out.texCoord.x = 0.5 * (1 + Pos.x + NPR_DilateErode_Erode_Vertex_Shader_pixelSize);
   Out.texCoord.y = 0.5 * (1 - Pos.y + NPR_DilateErode_Erode_Vertex_Shader_pixelSize);

   return Out;
}





float NPR_DilateErode_Erode_Pixel_Shader_pixelSize
<
   string UIName = "NPR_DilateErode_Erode_Pixel_Shader_pixelSize";
   string UIWidget = "Numeric";
   bool UIVisible =  true;
   float UIMin = 0.00;
   float UIMax = 0.01;
> = float( 0.00 );
sampler NPR_DilateErode_Erode_Pixel_Shader_ImageRT = sampler_state
{
   Texture = (ImageRT_Tex);
   ADDRESSU = CLAMP;
   ADDRESSV = CLAMP;
   MAGFILTER = LINEAR;
   MINFILTER = LINEAR;
   MIPFILTER = LINEAR;
};
// We are using reverse subtract blending, so the result will
// be dilate - erode, which is a basic edge-detecting operator.


// The surrounding pixels
float2 NPR_DilateErode_Erode_Pixel_Shader_samples[8] = {
   -1, -1,
    0, -1,
    1, -1,
   -1,  0,
    1,  0,
   -1,  1,
    0,  1,
    1,  1,
};

float4 NPR_DilateErode_Erode_Pixel_Shader_main(float2 texCoord: TEXCOORD) : COLOR {
   // Simple erode operation, find the minimum
   // pixel in the neighborhood.

   float4 minSamp = tex2D(NPR_DilateErode_Erode_Pixel_Shader_ImageRT, texCoord);
   for (int i = 0; i < 8; i++){
      float4 sample = tex2D(NPR_DilateErode_Erode_Pixel_Shader_ImageRT, texCoord + NPR_DilateErode_Erode_Pixel_Shader_pixelSize * NPR_DilateErode_Erode_Pixel_Shader_samples[i]);
      minSamp = min(minSamp, sample);
   }

   return minSamp;
}


//--------------------------------------------------------------//
// Technique Section for Effect Workspace.NPR.DilateErode
//--------------------------------------------------------------//
technique DilateErode
{
   pass DrawObject
   <
      string Script = "RenderColorTarget0 = ImageRT_Tex;"
                      "ClearColor = (0, 0, 0, 0);"
                      "ClearDepth = 1.000000;";
   >
   {
      ALPHABLENDENABLE = FALSE;

      VertexShader = compile vs_1_1 NPR_DilateErode_DrawObject_Vertex_Shader_main();
      PixelShader = compile ps_2_0 NPR_DilateErode_DrawObject_Pixel_Shader_main();
   }

   pass Dilate
   {
      CULLMODE = NONE;
      ALPHABLENDENABLE = FALSE;
      ZENABLE = FALSE;

      VertexShader = compile vs_1_1 NPR_DilateErode_Dilate_Vertex_Shader_main();
      PixelShader = compile ps_2_0 NPR_DilateErode_Dilate_Pixel_Shader_main();
   }

   pass Erode
   {
      SRCBLEND = ONE;
      DESTBLEND = ONE;
      CULLMODE = NONE;
      ALPHABLENDENABLE = TRUE;
      BLENDOP = REVSUBTRACT;

      VertexShader = compile vs_1_1 NPR_DilateErode_Erode_Vertex_Shader_main();
      PixelShader = compile ps_2_0 NPR_DilateErode_Erode_Pixel_Shader_main();
   }

}
And here's the .material I created (it should be obvious I haven't done this before)

Code: Select all

vertex_program hlslDrawWhiteVS hlsl
{
	source hlslDrawWhiteVS.txt
	entry_point main
	target vs_1_1
}

fragment_program hlslDrawWhitePS hlsl
{
	source hlslDrawWhitePS.txt
	entry_point main
	target ps_2_0
}

vertex_program hlslDilateVS hlsl
{
	source hlslDilateVS.txt
	entry_point main
	target vs_1_1
}

fragment_program hlslDilatePS hlsl
{
	source hlslDilatePS.txt
	entry_point main
	target ps_2_0
}

vertex_program hlslErodeVS hlsl
{
	source hlslErodeVS.txt
	entry_point main
	target vs_1_1
}

fragment_program hlslErodePS hlsl
{
	source hlslErodePS.txt
	entry_point main
	target ps_2_0
}

material DilateErodeShader
{
	technique
	{
		pass DrawWhite
		{
			depth_check off
			// scene_blend add is default

			vertex_program_ref hlslDrawWhiteVS
			{
			}

			fragment_program_ref hlslDrawWhitePS
			{
			}
			
			texture_unit RT
			{
                tex_coord_set 0
				tex_address_mode clamp
				filtering linear linear linear
			}
		}
		
		pass Dilate
		{
			depth_check off
			// depth_write off?
			cull_hardware none
			cull_software none

			vertex_program_ref hlslDilateVS
			{
			}

			fragment_program_ref hlslDilatePS
			{
			}
			
			texture_unit RT
			{
                tex_coord_set 0
				tex_address_mode clamp
				filtering linear linear linear
			}
		}
		
		pass Erode
		{
			depth_check off
			// depth_write off?
			cull_hardware none
			cull_software none
			scene_blend alpha_blend
			scene_blend_op reverse_subtract

			vertex_program_ref hlslErodeVS
			{
			}

			fragment_program_ref hlslErodePS
			{
			}
			
			texture_unit RT
			{
                tex_coord_set 0
				tex_address_mode clamp
				filtering linear linear linear
			}
		}
	}
}
Each .txt has the respective HLSL code in it.
At runtime if I use this material on my model there's no result (invisible model). Ogre.log doesn't have any complaints about my material.

I found this post here:
http://www.ogre3d.org/forums/viewtopic.php?f=5&t=35259
But the link, code, and images are all broken for me, so it's not very useful, but apparently this has been done before.

EDIT: I've identified some possible sources of error, so if anyone says "tl;dr" to the post maybe this will help:
- The "texture_unit RT" seems like it might only need to exist in the Dilate and Erode passes, not the DrawWhite, but despite reading the manual on this, I can't figure out what the crap this does.
- There's a lot of ambiguity between the .fx and the .material pass parameters, like "SRCBLEND" and stuff. I really don't know if I transferred them properly.

Post Reply