Tangent calculation error

Discussion area about developing or extending OGRE, adding plugins for it or building applications on it. No newbie questions please, use the Help forum for that.
Post Reply
User avatar
Kojack
OGRE Moderator
OGRE Moderator
Posts: 7157
Joined: Sun Jan 25, 2004 7:35 am
Location: Brisbane, Australia
x 534

Tangent calculation error

Post by Kojack »

I've been playing around with getting a high poly (around half a million tris) model from max into ogre (using easy ogre exporter). Everything is fine, it looks good.
That is until I tried applying a normal map and lighting shader. After lots of struggling (including several hours of debugging shaders before realising the hlsl compiler had optimised one of my texture samplers away, causing the specular map to shift into the normal map sampler), I've found that some of the tangents are being calculated as 0,0,0. These are valid triangles. Their positions and uv coords are fine, no degenerates or anything. This happens not only in Easy Ogre Exporter but also ogre's own Mesh::buildTangentVectors(), which I'm guessing EOE uses.

The cause seems to be triangles with small uv coord changes. The code that calculates the tangent and binormal vectors scales them by the area of the triangle. These scaled vectors are accumulated on a vertex to find the final tangent and binormal. But then it checks if either vector is zero length (using Vector3::isZeroLength), any zero length vectors are skipped, leaving the vertex with a blank tangent.
The problem is that the uv coords on the bad triangles have a delta of around 0.001 or smaller. This means the area comes out as under 1e-6. The isZeroLength method checks for zero length by testing if the vector's squared length is less than 1e-12. Since the tangent and binormal were scaled by the area (less than 1e-6) that means the squared length is less than 1e-12.

The uvs may sound too small, but this model is using 4096x4096 textures. On them, each pixel has a uv width of 2.4e-4. Even a 1024x1024 textures have a pixel width just below 1e-3. It's easy to have small triangles on such a high res texture, but ogre will reject them and break the tangents.

Is the area really supposed to scale the tangents? That seems a bit odd. I can imagine scaling by the angle between the vectors, but area seems wrong.

I found an article (while looking for alternative tangent code) that's very interesting: http://www.thetenthplanet.de/archives/1180
It shows how to do normal mapping with no tangents in the vertices. It doesn't even use tangents and binormals, it uses cotangents and cobinormals instead, calculated on the fly in the pixel shader. Apparently it can handle non orthogonal tangent spaces, which the regular code can't. I've converted it to hlsl and it seems to work well, but I don't know the performance hit.
User avatar
spacegaier
OGRE Team Member
OGRE Team Member
Posts: 4304
Joined: Mon Feb 04, 2008 2:02 pm
Location: Germany
x 135
Contact:

Re: Tangent calculation error

Post by spacegaier »

In addition to this discussion thread, this might be worth a JIRA ticket.
Ogre Admin [Admin, Dev, PR, Finance, Wiki, etc.] | BasicOgreFramework | AdvancedOgreFramework
Don't know what to do in your spare time? Help the Ogre wiki grow! Or squash a bug...
scrawl
OGRE Expert User
OGRE Expert User
Posts: 1119
Joined: Sat Jan 01, 2011 7:57 pm
x 216

Re: Tangent calculation error

Post by scrawl »

It does sound really weird that UVs would affect the tangents at all.
User avatar
Kojack
OGRE Moderator
OGRE Moderator
Posts: 7157
Joined: Sun Jan 25, 2004 7:35 am
Location: Brisbane, Australia
x 534

Re: Tangent calculation error

Post by Kojack »

spacegaier wrote:In addition to this discussion thread, this might be worth a JIRA ticket.
Yep. I was going to wait to see what the responses here were, since I'm not really sure where the problem is. Removing the "very small but not really zero" vector check did seem to fix the tangents, at least partly. I still got ones that looked bad around the corners of the mouth and eyes (the model is a female character). But maybe the area is the wrong way to scale the tangent and binormal before accumulating. I don't know if the artist wants the model shown (it's from a co-worker), so I need to make a new simple model that shows off the flaw.
scrawl wrote:It does sound really weird that UVs would affect the tangents at all.
Tangents are created by uvs. The normal, tangent and binormal form a 3d coordinate space that matches the flow of the uv coordinates along the surface.
User avatar
mkultra333
Gold Sponsor
Gold Sponsor
Posts: 1894
Joined: Sun Mar 08, 2009 5:25 am
x 114

Re: Tangent calculation error

Post by mkultra333 »

I've recently been looking into other ways to create tangent vectors for my map, because I want to store normal and tangent compressed in the Byte4 diffuse and spec colour of the mesh to save space. If I do that, I can't use ogre to generate the tangents. I looked around and found this code from Crytek.

http://www.crytek.com/cryengine/present ... alculation
http://www.crytek.com/download/Triangle ... lation.pdf

I mention this because their method also takes triangle size into consideration. Quoting from the pdf, they "Weight by the UV triangle size to avoid domination of small triangles."
"In theory there is no difference between practice and theory. In practice, there is." - Psychology Textbook.
User avatar
Kojack
OGRE Moderator
OGRE Moderator
Posts: 7157
Joined: Sun Jan 25, 2004 7:35 am
Location: Brisbane, Australia
x 534

Re: Tangent calculation error

Post by Kojack »

Using the technique from the page I linked, you'd be able to get away with just the normal stored in the mesh, but still have full tangent space normal mapping.

Here's part of the code, converted to hlsl (which means it's probably 99.99% cg compatible as well):

Code: Select all

float3x3 cotangent_frame( float3 N, float3 p, float2 uv )
{
    // get edge vectors of the pixel triangle
    float3 dp1 = ddx( p );
    float3 dp2 = ddy( p );
    float2 duv1 = ddx( uv );
    float2 duv2 = ddy( uv );
 
    // solve the linear system
    float3 dp2perp = cross( dp2, N );
    float3 dp1perp = cross( N, dp1 );
    float3 T = dp2perp * duv1.x + dp1perp * duv2.x;
    float3 B = dp2perp * duv1.y + dp1perp * duv2.y;
 
    // construct a scale-invariant frame 
    float invmax = rsqrt( max( dot(T,T), dot(B,B) ) );
    return float3x3( T * invmax, B * invmax, N );
}

// later in the main pixel shader:
float4 normalTex = tex2D(samplerNormal, uv)* 255.0/127.0 - 128.0/127.0;
normalTex.y = -normalTex.y;
float3x3 TBN = cotangent_frame( norm, -cameraPos, uv );
norm = normalize( mul(normalTex.xyz,TBN) );
I may have broken it a bit though while porting. I need to check with an already working model.
User avatar
mkultra333
Gold Sponsor
Gold Sponsor
Posts: 1894
Joined: Sun Mar 08, 2009 5:25 am
x 114

Re: Tangent calculation error

Post by mkultra333 »

Had a look at that article, sounds interesting and would be great if it works. I could replace two float3 with a single byte4, that's a saving of 20 bytes per vertex, which adds up when there are millions of vertices.

But is there any issue with the baked normal map texture applied to the model not using the same formula as the shader? The first few comments in the article mention this.

[Edit: Another thing, what's the license of the code he publishes? I hate that, online code with no license... drives me mad.]
"In theory there is no difference between practice and theory. In practice, there is." - Psychology Textbook.
User avatar
Kojack
OGRE Moderator
OGRE Moderator
Posts: 7157
Joined: Sun Jan 25, 2004 7:35 am
Location: Brisbane, Australia
x 534

Re: Tangent calculation error

Post by Kojack »

Here's ogre's tangent calculation:
Image

Here's the in-shader algorithm:
Image

In both cases, the tangent is displayed as abs(normalize(tangent)). So no pixel should ever be black.

The model didn't have smoothing groups on in max (it was actually made for a vray render, I think vray's materials dealt with that separately, because it renders smooth), that's why the tangents are faceted around the neck and forehead.
Post Reply