Issue with cel shading in 2 pass

Problems building or running the engine, queries about how to use features etc.
Post Reply
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Issue with cel shading in 2 pass

Post by gandf »

Hello,

I want to modify a material found on this forum to make a toon effect.

I would like to apply shader (cel shading) to texture.

To begin, I apply this material to ogre head on example scene. I have been using 2 pass because the Earring's material not done a good result.
If I use a 1 pass material, effect works but with this 2 pass material, a unstable black texture show up.

I am a beginner with shader and I don't understand why I have this result.

cg file :

Code: Select all

void main_vp(float4 position : POSITION, 
			 float3 normal : NORMAL, 
			 // outputs 
			 out float4 oPosition : POSITION, 
			 float2 texCoord3 : TEXCOORD0, 
			 out float specular : TEXCOORD1, 
			 out float edge : TEXCOORD2, 
			 out float2 uv3 : TEXCOORD3, 
			 // parameters 
			 uniform float3 lightPosition, // object space 
			 uniform float3 eyePosition, // object space 
			 uniform float4 shininess, 
			 uniform float4x4 worldViewProj) 
{ 
	uv3 = texCoord3;
	// calculate output position 
	oPosition = mul(worldViewProj, position); 

	// calculate light vector 
	float3 N = normalize(normal); 
	float3 L = normalize(lightPosition - position.xyz); 

	// Calculate diffuse component 
	//diffuse = max(dot(N, L) , 0);

	// Calculate specular component 
	float3 E = normalize(eyePosition - position.xyz); 
	float3 H = normalize(L + E); 
	specular = pow(max(dot(N, H), 0), shininess); 
	// Mask off specular if diffuse is 0 
	//if (diffuse == 0) specular = 0; 

	// Edge detection, dot eye and normal vectors 
	edge = max(dot(N, E), 0); 
} 

void main_fp(float diffuseIn : TEXCOORD0, 
			 float specularIn : TEXCOORD1, 
			 float edge : TEXCOORD2, 
			 float2 uv3: TEXCOORD3,

			 out float4 colour : COLOR, 

			 uniform float4 diffuse, 
			 uniform float4 specular, 

			 uniform sampler1D diffuseRamp, 
			 uniform sampler1D specularRamp, 
			 uniform sampler1D edgeRamp,
			 uniform sampler2D texture) 
{ 
	// Step functions from textures 
	diffuseIn = tex1D(diffuseRamp, diffuseIn).x; 
	specularIn = tex1D(specularRamp, specularIn).x; 
	edge = tex1D(edgeRamp, edge).x; 

	float4 texture_color = tex2D(texture, uv3); 
	colour = edge * ((texture_color * diffuseIn) + (texture_color * specularIn)); 
}

Code: Select all

// -------------------------------
// Cel Shading Section
// -------------------------------
vertex_program Ogre/CelShadingVP cg
{
	source Example_CelShading.cg
	entry_point main_vp
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto lightPosition light_position_object_space 0
		param_named_auto eyePosition camera_position_object_space
		param_named_auto worldViewProj worldviewproj_matrix
		param_named shininess float 10 
	}
}

fragment_program Ogre/CelShadingFP cg
{
	source Example_CelShading.cg
	entry_point main_fp
	profiles ps_1_1 arbfp1 fp20
}
material Ogre/Skin
{
	technique
	{
		pass
		{
			vertex_program_ref Ogre/CelShadingVP
			{
				// map shininess from custom renderable param 1
				param_named_auto shininess custom 1
			}

			ambient 0.3 0.8 0.3

			texture_unit
			{
				texture GreenSkin.jpg
				tex_address_mode mirror
			}
		}
		pass
		{
			fragment_program_ref Ogre/CelShadingFP
			{
				// map diffuse from custom renderable param 2
				param_named_auto diffuse custom 2
				// map specular from custom renderable param 2
				param_named_auto specular custom 3
			}

			texture_unit
			{
				texture cel_shading_diffuse.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit
			{
				texture cel_shading_specular.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit
			{
				texture cel_shading_edge.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit
			{
				colour_op_ex source2 src_texture src_texture
			}
		}
	}
}
result :
Image

Does anyone have a solution to offer me ?

Thanks
Last edited by gandf on Sun Mar 13, 2011 1:30 pm, edited 2 times in total.
StrakeFengala
Kobold
Posts: 28
Joined: Wed Feb 10, 2010 11:18 pm
Location: USA
x 4

Re: Issue with cel shading in 2 pass

Post by StrakeFengala »

Greetings,

[EDIT]
I'm not sure that the material I posted below would fix the earring problem, the ogrehead's earring is not even textured in the first place so you might want to stick with the original non-textured cel shading material for the earring while having the textured CelShading material for the rest of the head.

This is how the CelShading example does it, do this after setting the textured material for the head:

Code: Select all

SubEntity *sub = ent->getSubEntity(2);    // earring
sub->setMaterialName("Examples/CelShading"); //ADDED: You have to do this if you want to set the correct material for the earring after setting the textured CelShading material for the rest of the head.
sub->setCustomParameter(SP_SHININESS, Vector4(25, 0, 0, 0));
sub->setCustomParameter(SP_DIFFUSE, Vector4(1, 1, 0, 1));
sub->setCustomParameter(SP_SPECULAR, Vector4(1, 1, 0.7, 1));
[/EDIT]



I believe I have a solution, here is the CelShading material and program I use. It is based off of the method used in this forum page and the original CelShading material so it is not entirely my work :wink:. I edited it a little to add light attenuation and ambient light. It also supports maximum lights with iteration :).

Here is CelShading.cg

Code: Select all

void main_vp( float4 position : POSITION,
              float3 normal : NORMAL,
              // outputs
              out float4 oPosition : POSITION,
              out float  diffuse : TEXCOORD0,
              out float  specular : TEXCOORD1,
              out float  edge : TEXCOORD2,
              float2 texCoord3 : TEXCOORD0, 
              out float2 uv3 : TEXCOORD3,
              // parameters
              uniform float3 lightPosition, // object space
		      uniform float4 lightAttenuation,
              uniform float3 eyePosition,   // object space
              uniform float4  shininess,
              uniform float4x4 worldViewProj )
{
	// calculate output position
	oPosition = mul(worldViewProj, position);

	// calculate light vector
	float3 N = normalize(normal);
	float3 L = normalize(lightPosition - position.xyz);

	//Calculate distance from light to object.
	float lightDist = length( lightPosition - position.xyz );
	float luminosity = 0;
	if( lightAttenuation.x > lightDist)
		luminosity = 1.0 / ( lightAttenuation.y + lightAttenuation.z*lightDist + lightAttenuation.w*(lightDist*lightDist) );

	// Calculate diffuse component
	diffuse = max(dot(N, L) , 0)*luminosity;

	// Calculate specular component
	float3 E = normalize(eyePosition - position.xyz);
	float3 H = normalize(L + E);
	specular = pow(max(dot(N, H), 0), shininess)*luminosity;
	// Mask off specular if diffuse is 0
	if (diffuse == 0) specular = 0;

	// Edge detection, dot eye and normal vectors
	edge = max(dot(N, E), 0);
	uv3 = texCoord3;
}

void main_fp( float diffuseIn	: TEXCOORD0,
              float specularIn	: TEXCOORD1,
              float edge		: TEXCOORD2,
              
              out float4 colour	: COLOR,
              
              float2 uv3		: TEXCOORD3,
              
              uniform sampler1D diffuseRamp : register(s0),
              uniform sampler1D specularRamp : register(s2),
              uniform sampler1D edgeRamp : register(s1),
              uniform sampler2D diffmap : register(s3),
              uniform sampler2D specmap : register(s4) )
{
	// Step functions from textures
	diffuseIn = tex1D(diffuseRamp, diffuseIn).x;
	specularIn = tex1D(specularRamp, specularIn).x;
	edge = tex1D(edgeRamp, edge).x;

	float4 diffuse_map = tex2D( diffmap, uv3 );
	colour = edge * ( ( diffuse_map * (diffuseIn) ) + ( tex2D( specmap, uv3 ) * (specularIn)) ); 
	colour.a = diffuse_map.a; 
}

void ambientOneTexture_vp(float4 position : POSITION,
						  float2 uv		  : TEXCOORD0,
						  float3 normal   : NORMAL,
						  
						  out float4 oPosition : POSITION,
						  out float2 oUv	   : TEXCOORD0,
						  out float4 colour    : COLOR,
						  out float  edge      : TEXCOORD1,
						  out float4 oAmbient  : TEXCOORD2,

						  uniform float4x4 worldViewProj,
						  uniform float3 eyePosition,
						  uniform float4 ambient,
						  uniform sampler1D edgeRamp : register(s1))
{
	oPosition = mul(worldViewProj, position);
	oUv = uv;
	
	float3 N = normalize(normal);
	float3 E = normalize(eyePosition - position.xyz);
	edge = max(dot(N, E), 0);
	oAmbient = ambient;
	//colour = ambient;
}
void ambientOneTexture_fp(float  edge     : TEXCOORD1,
						  float4 ambient  : TEXCOORD2,
						  
						  out float4 colour    : COLOR,

						  uniform sampler1D edgeRamp : register(s1))
{
	edge = tex1D(edgeRamp, edge).x;
	colour = edge*ambient;
}
Now the CelShading.material file:

Code: Select all

vertex_program AmbientOneTextureVP cg
{
	source CelShading.cg
	
	default_params
	{
		param_named_auto worldViewProj worldviewproj_matrix
		param_named_auto ambient ambient_light_colour
		param_named_auto eyePosition camera_position_object_space
	}
	
	entry_point ambientOneTexture_vp
	profiles vs_1_1 arbvp1
}
fragment_program AmbientOneTextureFP cg
{
	source CelShading.cg
	entry_point ambientOneTexture_fp
	profiles ps_2_0 arbfp1 fp20
}
vertex_program CelShadingTexturedVP cg
{
   source CelShading.cg
   entry_point main_vp
   profiles vs_1_1 arbvp1

   default_params
   {
      param_named_auto lightPosition light_position_object_space 0
	  param_named_auto lightAttenuation light_attenuation 0
      param_named_auto eyePosition camera_position_object_space
      param_named_auto worldViewProj worldviewproj_matrix
      param_named shininess float 10 
   }
}

fragment_program CelShadingTexturedFP cg
{
   source CelShading.cg
   entry_point main_fp
   profiles ps_2_0 arbfp1 fp20
}

material CelShading
{
	technique
	{
		pass ambient
		{
			vertex_program_ref AmbientOneTextureVP
			{
			}
			fragment_program_ref AmbientOneTextureFP
			{
			}
			texture_unit amb_diff
			{
				texture diffuse_map.png
			}
			texture_unit
			{
				texture cel_shading_edge.png 1d
				tex_address_mode clamp
				filtering none
				tex_coord_set 1
			}
		}
		pass lighting
		{
			iteration once_per_light
			scene_blend add
			depth_write off

			vertex_program_ref CelShadingTexturedVP
			{
				param_named shininess float 1.0 // you can change this as well
			}
			fragment_program_ref CelShadingTexturedFP
			{
			}
			texture_unit
			{
				texture cel_shading_diffuse.png 1d
				tex_address_mode clamp
				filtering none
				colour_op add
			}
			texture_unit
			{
				texture cel_shading_edge.png 1d
				tex_address_mode clamp
				filtering none
				tex_coord_set 1
			}
			texture_unit
			{
				texture cel_shading_specular.png 1d
				tex_address_mode clamp
				filtering none
				tex_coord_set 2
				colour_op add
			}
			texture_unit diffMap
			{
				texture diffuse_map.png //Add your diffuse map here
			}
			texture_unit specMap
			{
				texture specular_map.png //Add you specular map here. You can comment out this entire section if you don't want specular
			}
		}
	}
}
Now in your own .material file for example, you can do something like this:

Code: Select all

import * from "CelShading.material"

material Ogre/Skin: CelShading
{
	technique
	{
		pass ambient
		{
			texture_unit amb_diff
			{
				texture GreenSkin.jpg
			}
		}
		pass lighting
		{
			texture_unit diffMap
			{
				texture GreenSkin.jpg
			}
			//You can completely comment out the next texture unit if you don't want specular lighting
			//Currently, there is no specular map for greenskin.jpg, not that it is impossible to make. Just de-saturate the image with something like GIMP and that's a start...
			//EDIT: I'm pretty sure you can just use a blank white image if you wanted specular all over. Or even edit CelShading.cg and replace "colour = edge * ( ( diffuse_map * (diffuseIn) ) + ( tex2D( specmap, uv3 ) * (specularIn)) );" with "colour = edge * ( ( diffuse_map * (diffuseIn) ) + specularIn );" taking out tex2D( specmap, uv3 ).
			//texture_unit specMap
			//{
			//	texture YourSpecularMap.blah
			//}
		}
	}
}
Hope this helps, let me know if anything goes wrong or if I made a mistake.

StrakeFengala
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

Really thanks. :D

I have been searching to make effect since last night but I have been blocking since yesterday.

Your code produce nearly the same effect that the code in 1 pass with add specular texture. All parameter outside texture are not taken into account.

I can do a 2 pass rendering with no problem on edge detect and light (1 pass or 2 pass).
But with 2 pass, I can't do to give texture from pass 1 to pass 2 and result in pass 1(texture + effect) is show simultaneously with result of pass 2.

result : I added a depth_check off effect to stop color mixing
Image

I tested this texture to pass 2 with no result :

Code: Select all

			texture_unit
			{
				scene_blend dest_colour dest_alpha
				//colour_op_ex source1 src_current
			}
I find this2 post but this code don't work.
http://www.ogre3d.org/forums/viewtopic. ... VP#p409170

Do you have another idea ?
StrakeFengala
Kobold
Posts: 28
Joined: Wed Feb 10, 2010 11:18 pm
Location: USA
x 4

Re: Issue with cel shading in 2 pass

Post by StrakeFengala »

I just noticed a problem in the shader and material I posted, I thought I tested it but I must have messed it up before I posted it :oops:, although this problem isn't exactly related to the earring problem, that's a different situation. Plus, I took out depth_write off and I don't have the earring sticking through the head like in the image you posted.

Anyway, I think I misunderstood your problem in the first post and I think I know how to fix it now. I'm working on a test application while fixing shader code and I'll try to get an example posted tomorrow.

[EDIT]: It's not tomorrow yet, but...

I was able to put together a cg and material file based off of the original CelShading material, (that means so far it does not have the benefits of dynamic ambient lights or light attenuation). Basically what I did was create two different types of materials: one textured: "CelShadingTex", and one without a texture: "CelShading."

Here is the CelShading.cg:

Code: Select all

/* Cel shading vertex program for single-pass rendering
   In this program, we want to calculate the diffuse and specular
   ramp components, and the edge factor (for doing simple outlining)
   For the outlining to look good, we need a pretty well curved model.
*/
void main_vp(float4 position	: POSITION,
			 float3 normal		: NORMAL,
			 // outputs
			 out float4 oPosition	 : POSITION,
			 out float  diffuse		 : TEXCOORD0,
			 out float  specular	 : TEXCOORD1,
			 out float  edge		 : TEXCOORD2,
			 // parameters
			 uniform float3 lightPosition, // object space
			 uniform float3 eyePosition,   // object space
			 uniform float4  shininess,
			 uniform float4x4 worldViewProj)
{
	// calculate output position
	oPosition = mul(worldViewProj, position);

	// calculate light vector
	float3 N = normalize(normal);
	float3 L = normalize(lightPosition - position.xyz);
	
	// Calculate diffuse component
	diffuse = max(dot(N, L) , 0);

	// Calculate specular component
	float3 E = normalize(eyePosition - position.xyz);
	float3 H = normalize(L + E);
	specular = pow(max(dot(N, H), 0), shininess);
	// Mask off specular if diffuse is 0
	if (diffuse == 0) specular = 0;

	// Edge detection, dot eye and normal vectors
	edge = max(dot(N, E), 0);
	
}

void main_fp(float diffuseIn 	: TEXCOORD0,
			 float specularIn	: TEXCOORD1,
			 float edge		: TEXCOORD2,
			 
			 out float4 colour	: COLOR,
			 
			 uniform float4 diffuse,
			 uniform float4 specular,
			 
			 uniform sampler1D diffuseRamp,
			 uniform sampler1D specularRamp,
			 uniform sampler1D edgeRamp)
{
	// Step functions from textures
	diffuseIn = tex1D(diffuseRamp, diffuseIn).x;
	specularIn = tex1D(specularRamp, specularIn).x;
	edge = tex1D(edgeRamp, edge).x;

	colour = edge * ((diffuse * diffuseIn) + 
					(specular * specularIn));
}

//========//
//Textured//
//========//
void mainTex_vp(float4 position	: POSITION,
			 float3 normal		: NORMAL,
			 float2 texCoord3 : TEXCOORD0, 
			 // outputs
			 out float4 oPosition : POSITION,
			 out float  diffuse		 : TEXCOORD0,
			 out float  specular	 : TEXCOORD1,
			 out float  edge		 : TEXCOORD2,
			 out float2	oUv			 : TEXCOORD3,
			 // parameters
			 uniform float3 lightPosition, // object space
			 uniform float3 eyePosition,   // object space
			 uniform float4  shininess,
			 uniform float4x4 worldViewProj,
			 uniform sampler2D diffMap)
{
	// calculate output position
	oPosition = mul(worldViewProj, position);

	// calculate light vector
	float3 N = normalize(normal);
	float3 L = normalize(lightPosition - position.xyz);
	
	// Calculate diffuse component
	diffuse = max(dot(N, L) , 0);

	// Calculate specular component
	float3 E = normalize(eyePosition - position.xyz);
	float3 H = normalize(L + E);
	specular = pow(max(dot(N, H), 0), shininess);
	// Mask off specular if diffuse is 0
	if (diffuse == 0) specular = 0;

	// Edge detection, dot eye and normal vectors
	edge = max(dot(N, E), 0);
	
	oUv = texCoord3;
}

void mainTex_fp(float diffuseIn 	: TEXCOORD0,
			 float specularIn	: TEXCOORD1,
			 float edge		: TEXCOORD2,
			 float2 uv       : TEXCOORD3,
			 
			 out float4 colour	: COLOR,
			 
			 uniform float4 diffuse,
			 uniform float4 specular,
			 
			 uniform sampler1D diffuseRamp : register(s0),
			 uniform sampler1D specularRamp : register(s1),
			 uniform sampler1D edgeRamp : register(s2),
			 uniform sampler2D diffMap : register(s3))
{
	// Step functions from textures
	diffuseIn = tex1D(diffuseRamp, diffuseIn).x;
	specularIn = tex1D(specularRamp, specularIn).x;
	edge = tex1D(edgeRamp, edge).x;
	
	float4 diffuseTex = tex2D( diffMap, uv );

	colour = edge * diffuseTex * ((diffuse * diffuseIn) + 
					(specular * specularIn));
	//colour = diffuseTex;
} 
And this CelChading.material file defines the base CelShading textured and non-textured materials.

Code: Select all

// -------------------------------
// Cel Shading Section
// -------------------------------
vertex_program CelShadingVP cg
{
	source CelShading.cg
	entry_point main_vp
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto lightPosition light_position_object_space 0
		param_named_auto eyePosition camera_position_object_space
		param_named_auto worldViewProj worldviewproj_matrix
		param_named shininess float 10 
	}
}

fragment_program CelShadingFP cg
{
	source CelShading.cg
	entry_point main_fp
	profiles ps_1_1 arbfp1 fp20
}

vertex_program CelShadingTexVP cg
{
	source CelShading.cg
	entry_point mainTex_vp
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto lightPosition light_position_object_space 0
		param_named_auto eyePosition camera_position_object_space
		param_named_auto worldViewProj worldviewproj_matrix
		param_named shininess float 10 
	}
}

fragment_program CelShadingTexFP cg
{
	source CelShading.cg
	entry_point mainTex_fp
	profiles ps_1_1 arbfp1 fp20
}


material CelShading
{
	technique
	{
		pass
		{
			vertex_program_ref CelShadingVP
			{
				// map shininess from custom renderable param 1
				param_named_auto shininess custom 1
			}
			fragment_program_ref CelShadingFP
			{
				// map diffuse from custom renderable param 2
				param_named_auto diffuse custom 2
				// map specular from custom renderable param 2
				param_named_auto specular custom 3
			}
			texture_unit
			{
				texture cel_shading_diffuse.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit
			{
				texture cel_shading_specular.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit
			{
				texture cel_shading_edge.png 1d
				tex_address_mode clamp
				filtering none
			}
		}
	}	
}
material CelShadingTex
{
	technique
	{
		pass
		{
			vertex_program_ref CelShadingTexVP
			{
				// map shininess from custom renderable param 1
				param_named_auto shininess float 25
			}
			fragment_program_ref CelShadingTexFP
			{
				// map diffuse from custom renderable param 2
				param_named_auto diffuse float4 1 1 1 1
				// map specular from custom renderable param 2
				param_named_auto specular float4 1 1 1 1
			}
			texture_unit
			{
				texture cel_shading_diffuse.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit
			{
				texture cel_shading_specular.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit
			{
				texture cel_shading_edge.png 1d
				tex_address_mode clamp
				filtering none
			}
			texture_unit diffMap
			{
				texture GreenSkin.jpg
			}
		}
	}	
}
This is another material file I use in my example program, it overrides the OgreHead's materials with the CelShading materials:

Code: Select all

import * from "CelShading.material"

material Ogre/Skin : CelShadingTex
{
	technique
	{
		pass
		{
			vertex_program_ref CelShadingTexVP
			{
				// map shininess from custom renderable param 1
				param_named shininess float 25
			}
			fragment_program_ref CelShadingTexFP
			{
				// map diffuse from custom renderable param 2
				param_named diffuse float4 1 1 1 1
				// map specular from custom renderable param 2
				param_named specular float4 0 0 0 1
			}
			texture_unit diffMap
			{
				texture GreenSkin.jpg
			}
		}
	}
}
material Ogre/Eyes : CelShading
{
	technique
	{
		pass
		{
			vertex_program_ref CelShadingVP
			{
				// map shininess from custom renderable param 1
				param_named shininess float 25
			}
			fragment_program_ref CelShadingFP
			{
				// map diffuse from custom renderable param 2
				param_named diffuse float4 1 0.7 0 1
				// map specular from custom renderable param 2
				param_named specular float4 0.3 0.1 0 1
			}
		}
	}
}
material Ogre/Tusks : CelShadingTex
{
	technique
	{
		pass
		{
			vertex_program_ref CelShadingTexVP
			{
				// map shininess from custom renderable param 1
				param_named shininess float 25
			}
			fragment_program_ref CelShadingTexFP
			{
				// map diffuse from custom renderable param 2
				param_named diffuse float4 0.7 0.7 0.6 1
				// map specular from custom renderable param 2
				param_named specular float4 0 0 0 1
			}
			texture_unit diffMap
			{
				texture tusk.jpg
			}
		}
	}
}
material Ogre/Earring : CelShading
{
	technique
	{
		pass
		{
			vertex_program_ref CelShadingVP
			{
				// map shininess from custom renderable param 1
				param_named shininess float 25
			}
			fragment_program_ref CelShadingFP
			{
				// map diffuse from custom renderable param 2
				param_named diffuse float4 0.7 0.7 0 1
				// map specular from custom renderable param 2
				param_named specular float4 1 1 1 1
			}
		}
	}
}
So far so good on my end, here is my result as an attachment:
OgreHead.jpg
Hope this helps.
[/EDIT][/b]
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

Thanks you. You have good skills shader.

I made ​​some changes to code for add an effect : color of edge is param.
To made this, I used ps_2_0 otherwise errors on compiling cg occured (too many arythmitic expression).

I have made an example to illustrate this effect.
edge of tusks : blue
edge of Earring : white
edge of others : black

I have add this effect to show a selected object in a scene point by mouse.

Here the picture to illustrate this effect :
Image

CelShading.cg :

Code: Select all

/* Cel shading vertex program for single-pass rendering
   In this program, we want to calculate the diffuse and specular
   ramp components, and the edge factor (for doing simple outlining)
   For the outlining to look good, we need a pretty well curved model.
*/
void main_vp(float4 position   : POSITION,
          float3 normal      : NORMAL,
          // outputs
          out float4 oPosition    : POSITION,
          out float  diffuse       : TEXCOORD0,
          out float  specular    : TEXCOORD1,
          out float  edge       : TEXCOORD2,
          // parameters
          uniform float3 lightPosition, // object space
          uniform float3 eyePosition,   // object space
          uniform float4  shininess,
          uniform float4x4 worldViewProj)
{
   // calculate output position
   oPosition = mul(worldViewProj, position);

   // calculate light vector
   float3 N = normalize(normal);
   float3 L = normalize(lightPosition - position.xyz);
   
   // Calculate diffuse component
   diffuse = max(dot(N, L) , 0);

   // Calculate specular component
   float3 E = normalize(eyePosition - position.xyz);
   float3 H = normalize(L + E);
   specular = pow(max(dot(N, H), 0), shininess);
   // Mask off specular if diffuse is 0
   if (diffuse == 0) specular = 0;

   // Edge detection, dot eye and normal vectors
   edge = max(dot(N, E), 0);
   
}

void main_fp(float diffuseIn    : TEXCOORD0,
          float specularIn   : TEXCOORD1,
          float edge      : TEXCOORD2,
          
          out float4 colour   : COLOR,
          
          uniform float4 diffuse,
          uniform float4 specular,
		  uniform float4 edgecolor,
          
          uniform sampler1D diffuseRamp,
          uniform sampler1D specularRamp,
          uniform sampler1D edgeRamp)
{
   // Step functions from textures
   diffuseIn = tex1D(diffuseRamp, diffuseIn).x;
   specularIn = tex1D(specularRamp, specularIn).x;
   edge = tex1D(edgeRamp, edge).x;
   
   colour = (1 - edge) * edgecolor + edge * ((diffuse * diffuseIn) + 
               (specular * specularIn));
}

//========//
//Textured//
//========//
void mainTex_vp(float4 position   : POSITION,
          float3 normal      : NORMAL,
          float2 texCoord3 : TEXCOORD0, 
          // outputs
          out float4 oPosition : POSITION,
          out float  diffuse       : TEXCOORD0,
          out float  specular    : TEXCOORD1,
          out float  edge       : TEXCOORD2,
          out float2   oUv          : TEXCOORD3,
          // parameters
          uniform float3 lightPosition, // object space
          uniform float3 eyePosition,   // object space
          uniform float4  shininess,
          uniform float4x4 worldViewProj,
          uniform sampler2D diffMap)
{
   // calculate output position
   oPosition = mul(worldViewProj, position);

   // calculate light vector
   float3 N = normalize(normal);
   float3 L = normalize(lightPosition - position.xyz);
   
   // Calculate diffuse component
   diffuse = max(dot(N, L) , 0);

   // Calculate specular component
   float3 E = normalize(eyePosition - position.xyz);
   float3 H = normalize(L + E);
   specular = pow(max(dot(N, H), 0), shininess);
   // Mask off specular if diffuse is 0
   if (diffuse == 0) specular = 0;

   // Edge detection, dot eye and normal vectors
   edge = max(dot(N, E), 0);
   
   oUv = texCoord3;
}

void mainTex_fp(float diffuseIn    : TEXCOORD0,
          float specularIn   : TEXCOORD1,
          float edge      : TEXCOORD2,
          float2 uv       : TEXCOORD3,
          
          out float4 colour   : COLOR,
          
          uniform float4 diffuse,
          uniform float4 specular,
		  uniform float4 edgecolor,
          
          uniform sampler1D diffuseRamp : register(s0),
          uniform sampler1D specularRamp : register(s1),
          uniform sampler1D edgeRamp : register(s2),
          uniform sampler2D diffMap : register(s3))
{
   // Step functions from textures
   diffuseIn = tex1D(diffuseRamp, diffuseIn).x;
   specularIn = tex1D(specularRamp, specularIn).x;
   edge = tex1D(edgeRamp, edge).x;
   
   float4 diffuseTex = tex2D( diffMap, uv );

   colour = (1 - edge) * edgecolor + edge * diffuseTex * ((diffuse * diffuseIn) + 
               (specular * specularIn));
} 
CelChading.material :

Code: Select all

// -------------------------------
// Cel Shading Section
// -------------------------------
vertex_program CelShadingVP cg
{
   source CelShading.cg
   entry_point main_vp
   profiles vs_1_1 arbvp1

   default_params
   {
      param_named_auto lightPosition light_position_object_space 0
      param_named_auto eyePosition camera_position_object_space
      param_named_auto worldViewProj worldviewproj_matrix
      param_named shininess float 10 
   }
}

fragment_program CelShadingFP cg
{
   source CelShading.cg
   entry_point main_fp
   profiles ps_2_0 arbfp1 fp20
}

vertex_program CelShadingTexVP cg
{
   source CelShading.cg
   entry_point mainTex_vp
   profiles vs_1_1 arbvp1

   default_params
   {
      param_named_auto lightPosition light_position_object_space 0
      param_named_auto eyePosition camera_position_object_space
      param_named_auto worldViewProj worldviewproj_matrix
      param_named shininess float 10 
   }
}

fragment_program CelShadingTexFP cg
{
   source CelShading.cg
   entry_point mainTex_fp
   profiles ps_2_0 arbfp1 fp20
}


material CelShading
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingVP
         {
            // map shininess from custom renderable param 1
            param_named_auto shininess custom 1
         }
         fragment_program_ref CelShadingFP
         {
            // map diffuse from custom renderable param 2
            param_named_auto diffuse custom 2
            // map specular from custom renderable param 2
            param_named_auto specular custom 3

			param_named edgecolor float4 0 0 0 0
         }
         texture_unit
         {
            texture cel_shading_diffuse.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_specular.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_edge.png 1d
            tex_address_mode clamp
            filtering none
         }
      }
   }   
}
material CelShadingTex
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingTexVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingTexFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 1 1 1 1
            // map specular from custom renderable param 2
            param_named specular float4 1 1 1 1

			param_named edgecolor float4 0 0 0 0
         }
         texture_unit
         {
            texture cel_shading_diffuse.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_specular.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_edge.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit diffMap
         {
            texture GreenSkin.jpg
         }
      }
   }   
}
Ogre.material :

Code: Select all

import * from "CelShading.material"

material Ogre/Skin: CelShadingTex
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingTexVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingTexFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 1 1 1 1
            // map specular from custom renderable param 2
            param_named specular float4 0 0 0 1

			param_named edgecolor float4 0 0 0 0

         }
         texture_unit diffMap
         {
            texture GreenSkin.jpg
         }
      }
   }
}
material Ogre/Eyes : CelShading
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 1 0.7 0 1
            // map specular from custom renderable param 2
            param_named specular float4 0.3 0.1 0 1

			param_named edgecolor float4 0 0 0 0
         }
      }
   }
}
material Ogre/Tusks : CelShadingTex
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingTexVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingTexFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 0.7 0.7 0.6 1
            // map specular from custom renderable param 2
            param_named specular float4 0 0 0 1

			param_named edgecolor float4 0 0.5 0.7 1
         }
         texture_unit diffMap
         {
            texture tusk.jpg
         }
      }
   }
}
material Ogre/Earring : CelShading
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 0.7 0.7 0 1
            // map specular from custom renderable param 2
            param_named specular float4 1 1 1 1

			param_named edgecolor float4 1 1 1 0
         }
      }
   }
}
Example supplied with ogre to illustrate cel shading is not broad enough and show a bad way to make. A designer are not a programmer. He may create a material however he will not compile the source program.
Your code should be integrated into an example and be delivered with Ogre3D.

Now, the last weakness of this material are the detection of edge.

Really thanks.
StrakeFengala
Kobold
Posts: 28
Joined: Wed Feb 10, 2010 11:18 pm
Location: USA
x 4

Re: [Solved] Issue with cel shading in 2 pass

Post by StrakeFengala »

I realize that this is labeled "Solved," but I would like to add a little more.

I do agree that the cel-shading example that comes with ogre is a bit lacking, but this is what it is: a fast, stable, basic example that doesn't need much resources to run. Also I like that edge color, it is a neat addition :).

Anyway, I made a few modifications to the shader code so that it is a little easier to handle with custom materials, (not to mention light attenuation, specular mapping, and dynamic ambient lighting are back in :D). With the addition of dynamic ambient you have to make sure that the left of your diffuse_ramp image is black, not grey.

I also added an shader snippet wiki page for those who may be interested: Enhanced CelShading.

Anyone, feel free to modify (but not mess up) the code if you have a better way to do this shader, I am learning too after all and I may not have coded the material the absolute best way.
User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
x 179
Contact:

Re: [Solved] Issue with cel shading in 2 pass

Post by jacmoe »

Really nice - thanks a lot! :)
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

Finally, the initial issue is resolved but the effect is not completed.

On the last screen, we can't see edge below mouth.
Algorithm to detect edge is weakness. I have found another algorithm but I don't know how to use it properly.
This algorithm is based on 'introduction to 3G game programming with direct X 9.0 (Frank D. Luna).
I converted the hlsl code to cg.

The questions are :
input are correct ?
When use the output ?

Thanks.

Celshading.cg

Code: Select all

/* Cel shading vertex program for single-pass rendering
   In this program, we want to calculate the diffuse and specular
   ramp components, and the edge factor (for doing simple outlining)
   For the outlining to look good, we need a pretty well curved model.
*/
void main_vp(float4 position   : POSITION,
          float3 normal      : NORMAL,
          // outputs
          out float4 oPosition    : POSITION,
          out float  diffuse       : TEXCOORD0,
          out float  specular    : TEXCOORD1,
          out float  edge       : TEXCOORD2,
          // parameters
          uniform float3 lightPosition, // object space
          uniform float3 eyePosition,   // object space
          uniform float4  shininess,
          uniform float4x4 worldViewProj)
{
   // calculate output position
   oPosition = mul(worldViewProj, position);

   // calculate light vector
   float3 N = normalize(normal);
   float3 L = normalize(lightPosition - position.xyz);
   
   // Calculate diffuse component
   diffuse = max(dot(N, L) , 0);

   // Calculate specular component
   float3 E = normalize(eyePosition - position.xyz);
   float3 H = normalize(L + E);
   specular = pow(max(dot(N, H), 0), shininess);
   // Mask off specular if diffuse is 0
   if (diffuse == 0) specular = 0;

   // Edge detection, dot eye and normal vectors
   edge = max(dot(N, E), 0);
   
}

void main_fp(float diffuseIn    : TEXCOORD0,
          float specularIn   : TEXCOORD1,
          float edge      : TEXCOORD2,
          
          out float4 colour   : COLOR,
          
          uniform float4 diffuse,
          uniform float4 specular,
		  uniform float4 edgecolor,
          
          uniform sampler1D diffuseRamp,
          uniform sampler1D specularRamp,
          uniform sampler1D edgeRamp)
{
   // Step functions from textures
   diffuseIn = tex1D(diffuseRamp, diffuseIn).x;
   specularIn = tex1D(specularRamp, specularIn).x;
   edge = tex1D(edgeRamp, edge).x;
   
   colour = (1 - edge) * edgecolor + edge * ((diffuse * diffuseIn) + 
               (specular * specularIn));
}

//*******************************************************************
static float4 Black = {0.0, 0.0, 0.0, 0.0};

struct VS_OUTPUT
{
	float3 position;
	float4 diffuse;
};

VS_OUTPUT Main(float4 position : POSITION,
				float4 normal : NORMAL0,
				float4 faceNormal1 : NORMAL1,
				float4 faceNormal2 : NORMAL2,
				uniform float4x4 worldViewProj,
				uniform float4x4 ProjMatrix)
{
	// zero out each member in output
	VS_OUTPUT output = (VS_OUTPUT)0;

	// transform position to view space
	position = mul(position, worldViewProj);

	// Compute a vector in the direction of the vertex
	// from the eye. Recall the eye is at the origin
	// in view space - eye is just camera position.
	float4 eyeToVertex = position;

	// transform normals to view space. Set w
	// components to zero since we're transforming vectors.
	// Assume there are no scalings in the world
	// matrix as well.
	normal.w = 0.0f;
	faceNormal1.w = 0.0f;
	faceNormal2.w = 0.0f;

	normal = mul(normal, worldViewProj);
	faceNormal1 = mul(faceNormal1, worldViewProj);
	faceNormal2 = mul(faceNormal2, worldViewProj);

	// compute the cosine of the angles between
	// the eyeToVertex vector and the face normals.
	float dot0 = dot(eyeToVertex, faceNormal1);
	float dot1 = dot(eyeToVertex, faceNormal2);

	// if cosines are different signs (positive/negative)
	// then we are on a silhouette edge. Do the signs
	// differ?
	if( (dot0 * dot1) < 0.0f )
	{
		// yes, then this vertex is on a silhouette edge,
		// offset the vertex position by some scalar in the
		// direction of the vertex normal.
		position += 0.1f * normal;
	}
	// transform to homogeneous clip space
	output.position = mul(position, ProjMatrix);

	// set outline color
	output.diffuse = Black;

	return output;
}

//========//
//Textured//
//========//
void mainTex_vp(float4 position   : POSITION,
          float3 normal      : NORMAL,
		  float3 faceNormal1 : NORMAL1,
		  float3 faceNormal2 : NORMAL2,
          float2 texCoord3 : TEXCOORD0, 
          // outputs
          out float4 oPosition : POSITION,
          out float  diffuse       : TEXCOORD0,
          out float  specular    : TEXCOORD1,
          out float  edge       : TEXCOORD2,
          out float2   oUv          : TEXCOORD3,
		  out float3 edgeposition,
		  out float4 edgediffuse,

          // parameters
          uniform float3 lightPosition, // object space
          uniform float3 eyePosition,   // object space
          uniform float4  shininess,
          uniform float4x4 worldViewProj,
		  uniform float4x4 viewProjectionMatrix,
          uniform sampler2D diffMap)
{
   // calculate output position
   oPosition = mul(worldViewProj, position);

   // calculate light vector
   float3 N = normalize(normal);
   float3 L = normalize(lightPosition - position.xyz);
   
   // Calculate diffuse component
   diffuse = max(dot(N, L) , 0);

   // Calculate specular component
   float3 E = normalize(eyePosition - position.xyz);
   float3 H = normalize(L + E);
   specular = pow(max(dot(N, H), 0), shininess);
   // Mask off specular if diffuse is 0
   if (diffuse == 0) specular = 0;

   // Edge detection, dot eye and normal vectors
   edge = max(dot(N, E), 0);

   VS_OUTPUT edge2 = Main(position, float4(normal, 0.0f), float4(faceNormal1, 0.0f), float4(faceNormal2, 0.0f), worldViewProj, viewProjectionMatrix);
   edgeposition = edge2.position;
   edgediffuse = edge2.diffuse;

   oUv = texCoord3;
}

void mainTex_fp(float diffuseIn    : TEXCOORD0,
          float specularIn   : TEXCOORD1,
          float edge      : TEXCOORD2,
          float2 uv       : TEXCOORD3,
		  float3 edgeposition,
		  float4 edgediffuse,
          
          out float4 colour   : COLOR,
          
          uniform float4 diffuse,
          uniform float4 specular,
		  uniform float4 edgecolor,
          
          uniform sampler1D diffuseRamp : register(s0),
          uniform sampler1D specularRamp : register(s1),
          uniform sampler1D edgeRamp : register(s2),
          uniform sampler2D diffMap : register(s3))
{
   // Step functions from textures
   diffuseIn = tex1D(diffuseRamp, diffuseIn).x;
   specularIn = tex1D(specularRamp, specularIn).x;
   edge = tex1D(edgeRamp, edge).x;
   
   float4 diffuseTex = tex2D( diffMap, uv );

   colour = (1 - edge) * edgecolor + edge * diffuseTex * ((diffuse * diffuseIn) + 
               (specular * specularIn));
} 

CelShading.material

Code: Select all

// -------------------------------
// Cel Shading Section
// -------------------------------
vertex_program CelShadingVP cg
{
   source CelShading.cg
   entry_point main_vp
   profiles vs_1_1 arbvp1

   default_params
   {
      param_named_auto lightPosition light_position_object_space 0
      param_named_auto eyePosition camera_position_object_space
      param_named_auto worldViewProj worldviewproj_matrix
      param_named shininess float 10 
   }
}

fragment_program CelShadingFP cg
{
   source CelShading.cg
   entry_point main_fp
   profiles ps_2_0 arbfp1 fp20
}

vertex_program CelShadingTexVP cg
{
   source CelShading.cg
   entry_point mainTex_vp
   profiles vs_1_1 arbvp1

   default_params
   {
      param_named_auto lightPosition light_position_object_space 0
      param_named_auto eyePosition camera_position_object_space
      param_named_auto worldViewProj worldviewproj_matrix
	  param_named_auto viewProjectionMatrix viewproj_matrix
      param_named shininess float 10 
   }
}

fragment_program CelShadingTexFP cg
{
   source CelShading.cg
   entry_point mainTex_fp
   profiles ps_2_0 arbfp1 fp20
}

material CelShading
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingVP
         {
            // map shininess from custom renderable param 1
            param_named_auto shininess custom 1
         }
         fragment_program_ref CelShadingFP
         {
            // map diffuse from custom renderable param 2
            param_named_auto diffuse custom 2
            // map specular from custom renderable param 2
            param_named_auto specular custom 3

			param_named edgecolor float4 0 0 0 0
         }
         texture_unit
         {
            texture cel_shading_diffuse.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_specular.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_edge.png 1d
            tex_address_mode clamp
            filtering none
         }
      }
   }   
}
material CelShadingTex
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingTexVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingTexFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 1 1 1 1
            // map specular from custom renderable param 2
            param_named specular float4 1 1 1 1

			param_named edgecolor float4 0 0 0 0
         }
         texture_unit
         {
            texture cel_shading_diffuse.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_specular.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit
         {
            texture cel_shading_edge.png 1d
            tex_address_mode clamp
            filtering none
         }
         texture_unit diffMap
         {
            texture GreenSkin.jpg
         }
      }
   }   
}
Ogre.material

Code: Select all

import * from "CelShading.material"

material Ogre/Skin: CelShadingTex
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingTexVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingTexFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 1 1 1 1
            // map specular from custom renderable param 2
            param_named specular float4 0 0 0 1

			param_named edgecolor float4 0 0 0 0

         }
         texture_unit diffMap
         {
            texture GreenSkin.jpg
         }
      }
   }
}
material Ogre/Eyes : CelShading
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 1 0.7 0 1
            // map specular from custom renderable param 2
            param_named specular float4 0.3 0.1 0 1

			param_named edgecolor float4 0 0 0 0
         }
      }
   }
}
material Ogre/Tusks : CelShadingTex
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingTexVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingTexFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 0.7 0.7 0.6 1
            // map specular from custom renderable param 2
            param_named specular float4 0 0 0 1

			param_named edgecolor float4 0 0 0 0
         }
         texture_unit diffMap
         {
            texture tusk.jpg
         }
      }
   }
}
material Ogre/Earring : CelShading
{
   technique
   {
      pass
      {
         vertex_program_ref CelShadingVP
         {
            // map shininess from custom renderable param 1
            param_named shininess float 25
         }
         fragment_program_ref CelShadingFP
         {
            // map diffuse from custom renderable param 2
            param_named diffuse float4 0.7 0.7 0 1
            // map specular from custom renderable param 2
            param_named specular float4 1 1 1 1

			param_named edgecolor float4 0 0 0 0
         }
      }
   }
}
StrakeFengala
Kobold
Posts: 28
Joined: Wed Feb 10, 2010 11:18 pm
Location: USA
x 4

Re: Issue with cel shading in 2 pass

Post by StrakeFengala »

I agree the edge detection is rather weak and looks best on smooth and high-poly models, but it is a simple edge detection that doesn't hog resources and looks ok for a handful of different things.

I don't know much shader coding beyond what I have already done, sorry. I think there is another way to do outline shading using RTT (Render to Texture) shaders which put REAL outlines around mesh edges (correct me if I'm wrong), though I'm sure there would be a definite framerate drop.

Before using Ogre, I have used a game development kit called DarkGDK but it handles things different to Ogre, apparently there are "Fullscreen Shaders" that I think is very similar to RTT. I used a cartoon outline fullscreen shader and applied it to the camera but I don't think it is that easy in Ogre. For an example of what I mean here, blender has built in edge detection and it looks really nice: http://www.blender.org/documentation/ht ... Toon02.png.

I haven't a clue on how to use RTT in Ogre at the moment, I just thought I'd mention this. I don't think I should post the HLSL code for the outline shader because I am unaware of it's copyright instructions since it came in a shader bundle with a program I bought, and it doesn't have any legal/copyright info inside the HLSL file itself. You might be able to find one somewhere else though.

I will still take a look at the new code you posted and see what I can make of it, and I'll post a result if I manage to get anywhere. :wink:
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

I think that we can do a shader to all scene with compitor and target in material file but we don't choose a different effet by objet.

The new code is "VS_OUTPUT Main".
He take vertices' normal to compute if the edge beetween them must be draw.
I don't know how we can give this information to program.

You have been doing a good job on wiki :D
StrakeFengala
Kobold
Posts: 28
Joined: Wed Feb 10, 2010 11:18 pm
Location: USA
x 4

Re: Issue with cel shading in 2 pass

Post by StrakeFengala »

Thanks for the compliment :D,

I have tried the code you posted, and I tried making a few adjustments but I could not get a "proper" result. What this code is supposed to do, I think, is to take vertices that are on the edges of the mesh and extrude them. If it is done this way it has to be done in at least two passes: edge outline, and then cel shading. The edge outline code is done with only a vertex program, no fragment program is needed.

On to the adjustments I made. According to this manual page, in the function mul(), the vector and matrix parameters have to be switched around if the vector is the first parameter.

The next part that I got messed up on is output.position = mul(ProjMatrix, position);
You see, the variable position is at first multiplied by the "World, View, and Projection" matrix and in the line output.position = mul(ProjMatrix, position); it is multiplied by the projection matrix again resulting in a weird image. I suspect that position should instead at first be multiplied by worldView instead of worldViewProj (for example:position = mul(worldView, position); the same thing I think, should go for the normal, facenormal1, and facenormal2 variables.

Even after this, I still got a strange output: black ogre head floating detached around the cel-shaded ogre head.

Here is the modified code:

Edge.cg

Code: Select all


void EdgeVP(float4 position : POSITION,
                   float4 normal : NORMAL0,
                   float4 faceNormal1 : NORMAL1,
                   float4 faceNormal2 : NORMAL2,
                   uniform float4x4 worldView,
                   uniform float4x4 ProjMatrix,
                   out float4 oColor : COLOR,
                   out float4 oPosition : POSITION)
{
   // transform position to view space
   position = mul(worldView, position);

   // Compute a vector in the direction of the vertex
   // from the eye. Recall the eye is at the origin
   // in view space - eye is just camera position.
   float4 eyeToVertex = position;

   // transform normals to view space. Set w
   // components to zero since we're transforming vectors.
   // Assume there are no scalings in the world
   // matrix as well.
   normal.w = 0.0f;
   faceNormal1.w = 0.0f;
   faceNormal2.w = 0.0f;

   normal = mul(worldView, normal);
   faceNormal1 = mul(worldView, faceNormal1);
   faceNormal2 = mul(worldView, faceNormal2);

   // compute the cosine of the angles between
   // the eyeToVertex vector and the face normals.
   float dot0 = dot(eyeToVertex, faceNormal1); //<-- For some reason, this always turns out to be 0, I suspect a problem.
   float dot1 = dot(eyeToVertex, faceNormal2);

   // if cosines are different signs (positive/negative)
   // then we are on a silhouette edge. Do the signs
   // differ?
   if( (dot0 * dot1) < 0.0f )
   {
      // yes, then this vertex is on a silhouette edge,
      // offset the vertex position by some scalar in the
      // direction of the vertex normal.
      position += 0.1f * normal;
   }
   // transform to homogeneous clip space
   oPosition = mul(ProjMatrix, position);

   // set outline color
   oColor = float4(0,0,0,1);//<-- same as Black

   return output;
}
Test material:

Code: Select all

vertex_program EdgeVP cg
{
	source Edge.cg
	entry_point EdgeVP
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto ProjMatrix projection_matrix
		param_named_auto worldView worldview_matrix
	}
}

material Ogre/Skin
{
   technique
   {
      pass edge
      {
         vertex_program_ref EdgeVP
         {
         }
      }
      pass Decal
      {
         texture_unit
         {
            texture GreenSkin.jpg
         }
      }
   }
}
Thanks for the tip about compositors, now that I read about RTT it's not entirely what I thought it was even though I knew it could "Render to texture," (I think I was getting the two mixed up :oops:). I think I will experiment with simple compositors and try a few things, NVIDIA has a shader called "scene lineDraw," in the NVIDIA Shader Library although it is way past my knowledge right now. I don't see any more modifications I can make to the code above at the moment, I'm going to rest my brain a bit. :?
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

I do not know how used this code but your EdgeVP must return no value when he returns output.
I had not seen this technical details for mul, sorry.
Result are like first image on my first post with less black texture.

On nvidia website, another shader is present but with no picture : scene toonEdges.
nvidia FX composer bug on my pc and I could not test it.
StrakeFengala
Kobold
Posts: 28
Joined: Wed Feb 10, 2010 11:18 pm
Location: USA
x 4

Re: Issue with cel shading in 2 pass

Post by StrakeFengala »

gandf wrote:I do not know how used this code but your EdgeVP must return no value when he returns output.
Sorry, I forgot to explain that I changed it so that it does not need to return an output. Instead it just sends to the COLOR and POSITION values which pretty much do the same thing.
StrakeFengala wrote:Even after this, I still got a strange output: black ogre head floating detached around the cel-shaded ogre head.
Sorry about that, that is wrong. I got mixed up again, this was a result of some other wild experimenting with the shader... :lol:

I tested again and got your result:
gandf wrote:Result are like first image on my first post with less black texture.
That will happen, (I would address that problem if this next problem can be solved), the real problem is that it wasn't extruding the edges anywhere:
StrakeFengala wrote:

Code: Select all

// compute the cosine of the angles between
// the eyeToVertex vector and the face normals.

[b]float dot0 = dot(eyeToVertex, faceNormal1); //<-- For some reason, this always turns out to be 0, I suspect a problem.[/b]

float dot1 = dot(eyeToVertex, faceNormal2);
I commented out "if( (dot0 * dot1) < 0.0f )" and then ALL of the vertices extruded (as to be expected).
StrakeFengala
Kobold
Posts: 28
Joined: Wed Feb 10, 2010 11:18 pm
Location: USA
x 4

Re: Issue with cel shading in 2 pass

Post by StrakeFengala »

Sorry for the double post, the previous one is a bit long and I'm about to post another long-ish one. :roll:

As I researched, some people were able to put an edge around a mesh by making a black copy, scaling it (or transforming vertices across their normals), and then flipping the normals. This would put a nice edge around the mesh, but it would not allow good edges within the mesh area.

I was able to write a vertex shader sort of based on that technique, this is more of a workaround rather than a solution though as there are slightly noticeable annoyances. What it does is transform the vertices directly away from the camera a little bit, and then extrude them along their normals, but we do not flip the normals afterward. It's very simple. It doesn't do a perfect job, a compositor with edge detect would still look much better.

Anyway, here's VertEdge.cg

Code: Select all

//*******************************************************************
void EdgeVP(float4 position : POSITION,
			float4 normal : NORMAL0,
			out float4 oPos : POSITION,
			out float4 oColor : COLOR,
			uniform float4 edgeColor,
			uniform float4 eyePosition,
			uniform float scale,
			uniform float edgeScale,
			uniform float4x4 worldViewProj)
{
	float4 E = normalize(eyePosition);
	position = mul(worldViewProj, position - scale*E);
	
	normal.w = 0;
	normal = normalize(mul(worldViewProj, normal));
	
	//     Transform vertices along their normals but do
	// not send them further away from the mesh.
	//     This explains why I use float4(normal.xy, 0, 0)
	// instead of just plain "normal."
	position += ((scale/8.0f)+1.0) * edgeScale * float4(normal.xy, 0, 0);
	
	oPos=position;
	oColor=edgeColor;
}
I will explain what "uniform float scale" does in a moment. [EDIT]: Added parameter "uniform float edgeScale," so now you can also adjust the size of the edge. If it is "1.0," it is its original size, if it is "2.0," it is twice its original size.

And now changes are made to my CelShading.material script:

Code: Select all

// A really basic ambient pass program, support for one texture coodinate set
vertex_program AmbientOneTexture cg
{
	source CelShading.cg
	entry_point ambientOneTexture_vp
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto worldViewProj worldviewproj_matrix
		param_named_auto ambient ambient_light_colour
		param_named diffuse float4 1 1 1 1
	}
}
vertex_program CelEdgeVP cg
{
	source CelShading.cg
	entry_point celEdge_vp
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto worldViewProj worldviewproj_matrix
		param_named_auto eyePosition camera_position_object_space
	}
	
}
fragment_program CelEdgeFP cg
{
	source CelShading.cg
	entry_point celEdge_fp
	profiles ps_1_1 arbfp1 fp20

	default_params
	{
		param_named edgeColor float4 0 0 0 1
	}
	
}
vertex_program VertEdge cg
{
	source VertEdge.cg
	entry_point EdgeVP
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto worldViewProj worldviewproj_matrix
		param_named_auto eyePosition camera_position_object_space
		param_named scale float 2
		param_named edgeScale float 1 //<--EDIT: Added this parameter for changing the size of the edge, defaults at 1.
	}
	
}
vertex_program CelShadingVP cg
{
	source CelShading.cg
	entry_point lighting_vp
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto lightPosition light_position_object_space 0
		param_named_auto lightAttenuation light_attenuation 0
		param_named_auto worldViewProj worldviewproj_matrix
	}
}
fragment_program CelShadingFP cg
{
	source CelShading.cg
	entry_point lighting_fp
	profiles ps_1_1 arbfp1 fp20
	
	default_params
	{
		param_named diffuse float4 1 1 1 1
	}
}
vertex_program CelSpecularVP cg
{
	source CelShading.cg
	entry_point specular_vp
	profiles vs_1_1 arbvp1

	default_params
	{
		param_named_auto lightPosition light_position_object_space 0
		param_named_auto lightAttenuation light_attenuation 0
		param_named_auto eyePosition camera_position_object_space
		param_named_auto worldViewProj worldviewproj_matrix
		param_named shininess float 10 
	}
}
fragment_program CelSpecularFP cg
{
	source CelShading.cg
	entry_point specular_fp
	profiles ps_1_1 arbfp1 fp20
}
fragment_program CelSpecularTexFP cg
{
	source CelShading.cg
	entry_point specularTex_fp
	profiles ps_1_1 arbfp1 fp20
}

material CelShading
{
	technique
	{
		set $diffuse "1 1 1 1"
		set $specular "1 1 1 1"
		set $edge "0 0 0 1"
		set $shininess "25"
		set $modscale "2"
		set $edgescale "1" //<--New variable to change the edge scale from other materials inheriting this one easily.
		
		pass edge
		{
			vertex_program_ref VertEdge
			{
				param_named edgeColor float4 $edge
				param_named scale float $modscale
				param_named edgeScale float $edgescale
			}
		}
		// Base ambient pass
		pass ambient
		{
			// base colours, not needed for rendering, but as information
			// to lighting pass categorisation routine
			ambient 1 1 1
			diffuse 0 0 0 
			specular 0 0 0 0 
			// Really basic vertex program
			// NB we don't use fixed function here because GL does not like
			// mixing fixed function and vertex programs, depth fighting can
			// be an issue
			vertex_program_ref AmbientOneTexture
			{
			}
		}
		pass perlight
		{
			ambient 0 0 0
			iteration once_per_light
			scene_blend add
			
			vertex_program_ref CelShadingVP
			{
			}
			fragment_program_ref CelShadingFP
			{
			}
			texture_unit
			{
				texture cel_shading_diffuse.png 1d
				tex_address_mode clamp
				filtering trilinear
				colour_op replace
			}
		}
		pass decal
		{
			lighting off

			vertex_program_ref AmbientOneTexture
			{
				param_named_auto worldViewProj worldviewproj_matrix
				param_named ambient float4 1 1 1 1
				param_named diffuse float4 $diffuse
			}
			scene_blend dest_colour zero
			texture_unit decalmap
			{
			}
		}
		pass specular
		{
			ambient 0 0 0
			iteration once_per_light
			scene_blend add
			
			vertex_program_ref CelSpecularVP
			{
				// map shininess from custom renderable param 1
				param_named shininess float $shininess
			}
			fragment_program_ref CelSpecularFP
			{
				// map specular from custom renderable param 2
				param_named specular float4 $specular
			}
			texture_unit
			{
				texture cel_shading_specular.png 1d
				tex_address_mode clamp
				filtering trilinear
				colour_op replace
			}
		}
	}		
}
material CelShadingSpecular
{
	technique
	{
		set $diffuse "1 1 1 1"
		set $specular "1 1 1 1"
		set $edge "0 0 0 1"
		set $shininess "25"
		set $modscale "2"
		set $edgescale "1" //<--New variable to change the edge scale from other materials inheriting this one easily.

		pass edge
		{
			vertex_program_ref VertEdge
			{
				param_named edgeColor float4 $edge
				param_named scale float $modscale
				param_named edgeScale float $edgescale
			}
		}
		// Base ambient pass
		pass ambient
		{
			// base colours, not needed for rendering, but as information
			// to lighting pass categorisation routine
			ambient 1 1 1
			diffuse 0 0 0 
			specular 0 0 0 0 
			// Really basic vertex program
			// NB we don't use fixed function here because GL does not like
			// mixing fixed function and vertex programs, depth fighting can
			// be an issue
			vertex_program_ref AmbientOneTexture
			{
			}
		}
		pass perlight
		{
			ambient 0 0 0
			iteration once_per_light
			scene_blend add
			
			vertex_program_ref CelShadingVP
			{
			}
			fragment_program_ref CelShadingFP
			{
			}
			texture_unit
			{
				texture cel_shading_diffuse.png 1d
				tex_address_mode clamp
				filtering trilinear
				colour_op replace
			}
		}
		pass decal
		{
			lighting off

			vertex_program_ref AmbientOneTexture
			{
				param_named_auto worldViewProj worldviewproj_matrix
				param_named ambient float4 1 1 1 1
				param_named diffuse float4 $diffuse
			}
			scene_blend dest_colour zero
			texture_unit decalmap
			{
			}
		}
		pass specular
		{
			ambient 0 0 0
			iteration once_per_light
			scene_blend add
			
			vertex_program_ref CelSpecularVP
			{
				// map shininess from custom renderable param 1
				param_named shininess float $shininess
			}
			fragment_program_ref CelSpecularTexFP
			{
				// map specular from custom renderable param 2
				param_named specular float4 $specular
			}
			texture_unit
			{
				texture cel_shading_specular.png 1d
				tex_address_mode clamp
				filtering trilinear
				colour_op replace
			}
			texture_unit specMap
			{
				texture white.png
			}
		}
	}	
}
Notice the variable $modscale (short for "model scale"), the variable is transferred to the edge shader variable "uniform float scale." The larger the scale, the farther the vertices will transform away from the camera and the smoother your edges will be. But there is a problem with sending it too far with smaller meshes (like the earring) because then the vertices would go inside a bigger mesh it may be next to (like the actual head). The variable defaults at 2.0, which is good for the earrings/tusks but bad for the head. So in the example material below, we will set $modscale "5.0" so that the edges will be smoother, [EDIT]:I changed the $modscale of Ogre/Skin to "4.0" and it still looks good (a little better in fact).

And now for the example OgreStuff.material:

Code: Select all

import * from "CelShading.material"

material Ogre/Skin : CelShadingSpecular
{
	technique
	{
		set $specular "0.3 0.3 0.3 1"
		set $modscale "4.0" //<--Used to be 5.0, edited to 4.0 and still looks good. Tried 3, but it is not as good as 4, too many jagged triangles.

		pass decal
		{
			texture_unit decalmap
			{
				texture GreenSkin.jpg 
			}
		}
		pass specular
		{
			texture_unit specMap
			{
				texture GreenSkin.jpg 
			}
		}
	}
}
material Ogre/Eyes : CelShading
{
	technique
	{
		set $diffuse "1 0.7 0 1"
		set $specular "1 0.8 0 1"
		set $edge "0 0 0 0"
	}
}
material Ogre/Tusks : CelShading
{
	technique
	{
		set $diffuse "0.7 0.7 0.6 1"
		set $specular "0 0 0 1"
		set $edge "0.4 0.4 0.3 1"
		pass decal
		{
			texture_unit decalmap
			{
				texture tusk.jpg
			}
		}
	}
}
material Ogre/Earring : CelShading
{
	technique
	{
		set $diffuse "0.7 0.7 0 1"
		set $specular "1 1 1 1"
		set $edge "0.6 0.6 0 1"
	}
}
[EDIT AGAIN]:Oops, I forgot to take out that $ambient variable before I posted :oops:, it wasn't doing anything anyway (besides taking up a little memory). I was trying to add custom ambient coloring one time but I wasn't doing it right, and I left those useless variables in there. It's gone now :roll:.

Here is a result so that you can get an idea of what it is supposed to look like before you try it out yourself:
screeny.jpg
[EDIT]:The edges on the earring and tusks are colored, not messed up (just so you know), just for fun.

I'm not going to put it into the wiki (yet) because I don't know if it is good enough, I'm sure it could use some improvements. If you are willing to try it, let me know how it works out. :wink:

(By the way, in the texture_units for the diffuse and specular ramps, I switched filtering to "trilinear" instead of "none". I like that smooth-ish effect better, it's just a preference I guess).
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

It's better. Here, a picture :

Image

I have seen on http://www.ogre3d.org/docs/manual/manual_18.html#SEC104 this code :

Code: Select all

uses_adjacency_information true
With this directive, we can access to neighbors vertices with a geometry program. With this program, we can recover normal of vertices to feeds edgeVP with these data and use an algorithm better.

I search tomorrow how to do this program.
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

I found the following document :
http://developer.download.nvidia.com/pr ... aph-06.pdf

I started to write some code. It is not over. Nothing is tested.
A geometry shader return normal of each neighbor.

EdgeGS.cg

Code: Select all

//compute normal
float3[2] computeNormal( float4 v1, float4 v2, float4 vAdj, float3 Eye)
{
	float3[2] output;
	output[0] = normalize(cross( v1-v2, vAdj-v2 )); // normal
	output[1] = normalize( Eye - v1.xyz ); // eye
	return output;
}

TRIANGLE_ADJ
TRIANGLE_OUT

//GS shader
void EdgeBuildDataGS( AttribArray<float4> pos: POSITION,
						AttribArray<float4> normal: NORMAL,
		out float3[3] NormalOut,
		out float3[3] EyePosOut,
		uniform float3 eyePosition)
{ 
	//compute the triangle’s normal
	float3[3] Norm;
	float3[3] Eye;

	float3 N1= normalize(cross( pos[0] - pos[2],
								pos[4] - pos[2] ));
	float3 eyeVec = normalize(eyePosition – pos[0].xyz);
	
	//if the central triangle is front facing, check the other triangles
	if(dot(N1, eyeVec)> 0.0f )
	{
		// compute normal forearch
		float3[2]buffer = computeNormal(pos[2], pos[0], pos[1], eyePosition);
		Norm[0] = buffer[0];
		Eye[0] = buffer[1];

		buffer = computeNormal(pos[4], pos[2], pos[3], eyePosition);
		Norm[1] = buffer[0];
		Eye[1] = buffer[1];

		buffer = computeNormal(pos[0], pos[4], pos[5], eyePosition);
		Norm[2] = buffer[0];
		Eye[2] = buffer[1];
	}
	else
	{ // void result
		Norm[0] = 0;
		Norm[1] = 0;
		Norm[2] = 0;
		Eye[0] = 0;
		Eye[1] = 0;
		Eye[2] = 0;
	}
	NormalOut = Norm;
	EyePosOut = Eye;
}
CelShading.material.

Code: Select all

geometry_program CelEdgeBuild cg
{
   source EdgeGS.cg
   entry_point EdgeBuildDataGS
   profiles vs_3_0 arbvp1
   	uses_adjacency_information true

   default_params
   {
     // param_named_auto worldViewProj worldviewproj_matrix
      param_named_auto eyePosition camera_position_object_space
   }
}
It's the beginning. I have indicated a vs_3_0 but I don't know minimum version to indicate.

EDIT : I rewrote some code. It's not finished.

EDIT2 : I have an error on compil with "eyePosition". It's marked undefined. Do you know why ? : float3 eyeVec = normalize(eyePosition – pos[0].xyz);
22:07:47: OGRE EXCEPTION(7:InternalErrorException): Unable to compile Cg program CelEdgeBuild: CG ERROR : "The compile returned an error."
(26) : error C0000: syntax error, unexpected $undefined, expecting ')' at token "<undefined>"
(26) : error C0501: type name expected at token "<undefined>"
(30) : error C0000: syntax error, unexpected '{', expecting ')' at token "{"
(30) : error C0501: type name expected at token "{"
(32) : error C1066: invalid type in type constructor
(32) : error C0000: syntax error, unexpected ';', expecting ')' at token ";"
(32) : error C0501: type name expected at token ";"
(36) : error C1066: invalid type in type constructor
(36) : error C0000: syntax error, unexpected ';', expecting ')' at token ";"
(36) : error C0501: type name expected at token ";"
(40) : error C1066: invalid type in type constructor
(40) : error C0000: syntax error, unexpected ';', expecting ')' at token ";"
(40) : error C0501: type name expected at token ";"
I found the NORMAL input to geometry shader. Compute normal what is still necessary ?
Thanks.
gandf
Gnoblar
Posts: 24
Joined: Sun Mar 06, 2011 12:03 am

Re: Issue with cel shading in 2 pass

Post by gandf »

I have make a new edge detect program in a geometry shader.
can you confirm the validity of my method ?
I think we should perhaps look at the angle between normals and draw a line if the angle is greater than a certain value such as 45°.

This function take in input :
- points list of the triangle + points to make adjacent triangles
- eyePosition
- worldViewProj

Result : a table for the 3 external points of adjacent triangles.
float3[3] NormalOut, // normal to compute edge
float3[3] EyePosOut,
float3[3] ExternPoint, // point to find side
float[3] edge, // trace edge ?

Code: Select all

//compute normal
float3[2] computeNormal( float4 v1, float4 v2, float4 vAdj, float3 Eye)
{
	float3[2] output;
	output[0] = normalize(cross( v1-v2, vAdj-v2 )); // normal
	output[1] = normalize( Eye - v1.xyz ); // eye
	return output;
}

TRIANGLE_ADJ

//GS shader
void EdgeBuildDataGS( AttribArray<float4> pos: POSITION,
		out float3[3] NormalOut, // normal to compute edge
		out float3[3] EyePosOut,
		out float3[3] ExternPoint, // point to find side
		out float[3] edge, // trace edge ?
		uniform float3 eyePosition,
		uniform float4x4 worldViewProj)
{ 
	//compute the triangle’s normal

	//record side to compute
	ExternPoint[0] = pos[1].xyz;
	ExternPoint[1] = pos[3].xyz;
	ExternPoint[2] = pos[5].xyz;

	float3 N1= normalize(cross( pos[0] - pos[2],
								pos[4] - pos[2] ));
	float3 eyeVec = normalize(eyePosition – pos[0].xyz);
	
	//if the central triangle is front facing, check the other triangles
	if(dot(buffer[0], buffer[1])> 0.0f )
	{
		//triangle visible

		// transform normals to view space.
		N1 = mul(WorldViewMatrix, N1);
		// compute the cosine of the angles between
		// the eyeToVertex vector and the face normals.
		float dot0 = dot(eyeToVertex, N1);

		// compute normal forearch
		float3[2]buffer = computeNormal(pos[2], pos[0], pos[1], eyePosition);
		NormalOut[0] = buffer[0];
		EyePosOut[0] = buffer[1];
		float3 N2 = mul(WorldViewMatrix, buffer[0]);
		float dot1 = dot(eyeToVertex, N2);

		// if cosines are different signs (positive/negative)
		// then we are on a silhouette edge. Do the signs
		// differ?
		if( (dot0 * dot1) < 0.0f )
		{
			// yes, then this vertex is on a silhouette edge,
			// offset the vertex position by some scalar in the
			// direction of the vertex normal.
			edge[0] = 1;
		}
		else
		{
			edge[2] = 0;
		}

		buffer = computeNormal(pos[4], pos[2], pos[3], eyePosition);
		NormalOut[1] = buffer[0];
		EyePosOut[1] = buffer[1];
		N2 = mul(WorldViewMatrix, buffer[0]);
		float dot1 = dot(eyeToVertex, N2);
		// ...
		if( (dot0 * dot1) < 0.0f )
		{
			// ...
			edge[1] = 1;
		}
		else
		{
			edge[2] = 0;
		}

		buffer = computeNormal(pos[0], pos[4], pos[5], eyePosition);
		NormalOut[2] = buffer[0];
		EyePosOut[2] = buffer[1];
		N2 = mul(WorldViewMatrix, buffer[0]);
		float dot1 = dot(eyeToVertex, N2);
		// ...
		if( (dot0 * dot1) < 0.0f )
		{
			// ...
			edge[2] = 1;
		}
		else
		{
			edge[2] = 0;
		}
	}
	else
	{ // void result : triangle hidden
		NormalOut[0] = 0;
		NormalOut[1] = 0;
		NormalOut[2] = 0;
		EyePosOut[0] = 0;
		EyePosOut[1] = 0;
		EyePosOut[2] = 0;
		edge[0] = 0;
		edge[1] = 0;
		edge[2] = 0;
	}
}
EDIT :
I modified the program to calculate only what is necessary.
That is, the algorithm calculates whether the angle between two triangles is greater than 45 ° to determine whether to draw or not an edge.

Code: Select all

TRIANGLE_ADJ

//GS shader
void EdgeBuildDataGS( AttribArray<float4> pos: POSITION,
		out float3[3] NormalOut, // normal to compute edge
		out float3[3] Vert1, // point to find side
		out float3[3] Vert2, // point to find side
		out float[3] edge, // trace edge ?
		uniform float cosangle)
{ 
	//compute the triangle’s normal

	//record side to compute
	Vert1[0] = pos[0].xyz; // edge 1
	Vert2[0] = pos[2].xyz;
	Vert1[1] = pos[2].xyz; // edge 2
	Vert2[1] = pos[4].xyz;
	Vert1[2] = pos[0].xyz; // edge 3
	Vert2[2] = pos[4].xyz;

	// compute normal forearch
	NormalOut[0] = normalize(cross(pos[2] - pos[0], pos[1] - pos[0])); // normal
	NormalOut[1] = normalize(cross(pos[4] - pos[2], pos[3] - pos[2])); // normal
	NormalOut[2] = normalize(cross(pos[0] - pos[4], pos[5] - pos[4])); // normal

	// compute normal to calculate angle
	float3 N1= normalize(cross(pos[0] - pos[2], pos[4] - pos[2]));

	//compute angle
	if(dot(N1, NormalOut[0]) < cosangle)
	{edge[0] = 1;}
	else {edge[0] = 0;}

	if(dot(N1, NormalOut[1]) < cosangle)
	{edge[1] = 1;}
	else {edge[1] = 0;}

	if(dot(N1, NormalOut[2]) < cosangle)
	{edge[2] = 1;}
	else {edge[2] = 0;}
}
An error is always raised :
19:19:26: OGRE EXCEPTION(7:InternalErrorException): Unable to compile Cg program CelEdgeBuild: CG ERROR : "The compile returned an error."
(0) : error C3001: no program defined
in CgProgram::loadFromSource at ..\..\..\..\..\PlugIns\CgProgramManager\src\OgreCgProgramManagerDll.cpp (line 67)
19:19:26: High-level program CelEdgeBuild encountered an error during loading and is thus not supported.
Post Reply