Page 1 of 1

Tangent calculation error

Posted: Fri Nov 08, 2013 11:59 am
by Kojack
I've been playing around with getting a high poly (around half a million tris) model from max into ogre (using easy ogre exporter). Everything is fine, it looks good.
That is until I tried applying a normal map and lighting shader. After lots of struggling (including several hours of debugging shaders before realising the hlsl compiler had optimised one of my texture samplers away, causing the specular map to shift into the normal map sampler), I've found that some of the tangents are being calculated as 0,0,0. These are valid triangles. Their positions and uv coords are fine, no degenerates or anything. This happens not only in Easy Ogre Exporter but also ogre's own Mesh::buildTangentVectors(), which I'm guessing EOE uses.

The cause seems to be triangles with small uv coord changes. The code that calculates the tangent and binormal vectors scales them by the area of the triangle. These scaled vectors are accumulated on a vertex to find the final tangent and binormal. But then it checks if either vector is zero length (using Vector3::isZeroLength), any zero length vectors are skipped, leaving the vertex with a blank tangent.
The problem is that the uv coords on the bad triangles have a delta of around 0.001 or smaller. This means the area comes out as under 1e-6. The isZeroLength method checks for zero length by testing if the vector's squared length is less than 1e-12. Since the tangent and binormal were scaled by the area (less than 1e-6) that means the squared length is less than 1e-12.

The uvs may sound too small, but this model is using 4096x4096 textures. On them, each pixel has a uv width of 2.4e-4. Even a 1024x1024 textures have a pixel width just below 1e-3. It's easy to have small triangles on such a high res texture, but ogre will reject them and break the tangents.

Is the area really supposed to scale the tangents? That seems a bit odd. I can imagine scaling by the angle between the vectors, but area seems wrong.

I found an article (while looking for alternative tangent code) that's very interesting:
It shows how to do normal mapping with no tangents in the vertices. It doesn't even use tangents and binormals, it uses cotangents and cobinormals instead, calculated on the fly in the pixel shader. Apparently it can handle non orthogonal tangent spaces, which the regular code can't. I've converted it to hlsl and it seems to work well, but I don't know the performance hit.

Re: Tangent calculation error

Posted: Fri Nov 08, 2013 12:47 pm
by spacegaier
In addition to this discussion thread, this might be worth a JIRA ticket.

Re: Tangent calculation error

Posted: Fri Nov 08, 2013 12:54 pm
by scrawl
It does sound really weird that UVs would affect the tangents at all.

Re: Tangent calculation error

Posted: Fri Nov 08, 2013 1:19 pm
by Kojack
spacegaier wrote:In addition to this discussion thread, this might be worth a JIRA ticket.
Yep. I was going to wait to see what the responses here were, since I'm not really sure where the problem is. Removing the "very small but not really zero" vector check did seem to fix the tangents, at least partly. I still got ones that looked bad around the corners of the mouth and eyes (the model is a female character). But maybe the area is the wrong way to scale the tangent and binormal before accumulating. I don't know if the artist wants the model shown (it's from a co-worker), so I need to make a new simple model that shows off the flaw.
scrawl wrote:It does sound really weird that UVs would affect the tangents at all.
Tangents are created by uvs. The normal, tangent and binormal form a 3d coordinate space that matches the flow of the uv coordinates along the surface.

Re: Tangent calculation error

Posted: Fri Nov 08, 2013 3:14 pm
by mkultra333
I've recently been looking into other ways to create tangent vectors for my map, because I want to store normal and tangent compressed in the Byte4 diffuse and spec colour of the mesh to save space. If I do that, I can't use ogre to generate the tangents. I looked around and found this code from Crytek. ... alculation ... lation.pdf

I mention this because their method also takes triangle size into consideration. Quoting from the pdf, they "Weight by the UV triangle size to avoid domination of small triangles."

Re: Tangent calculation error

Posted: Fri Nov 08, 2013 3:21 pm
by Kojack
Using the technique from the page I linked, you'd be able to get away with just the normal stored in the mesh, but still have full tangent space normal mapping.

Here's part of the code, converted to hlsl (which means it's probably 99.99% cg compatible as well):

Code: Select all

float3x3 cotangent_frame( float3 N, float3 p, float2 uv )
    // get edge vectors of the pixel triangle
    float3 dp1 = ddx( p );
    float3 dp2 = ddy( p );
    float2 duv1 = ddx( uv );
    float2 duv2 = ddy( uv );
    // solve the linear system
    float3 dp2perp = cross( dp2, N );
    float3 dp1perp = cross( N, dp1 );
    float3 T = dp2perp * duv1.x + dp1perp * duv2.x;
    float3 B = dp2perp * duv1.y + dp1perp * duv2.y;
    // construct a scale-invariant frame 
    float invmax = rsqrt( max( dot(T,T), dot(B,B) ) );
    return float3x3( T * invmax, B * invmax, N );

// later in the main pixel shader:
float4 normalTex = tex2D(samplerNormal, uv)* 255.0/127.0 - 128.0/127.0;
normalTex.y = -normalTex.y;
float3x3 TBN = cotangent_frame( norm, -cameraPos, uv );
norm = normalize( mul(,TBN) );
I may have broken it a bit though while porting. I need to check with an already working model.

Re: Tangent calculation error

Posted: Fri Nov 08, 2013 3:45 pm
by mkultra333
Had a look at that article, sounds interesting and would be great if it works. I could replace two float3 with a single byte4, that's a saving of 20 bytes per vertex, which adds up when there are millions of vertices.

But is there any issue with the baked normal map texture applied to the model not using the same formula as the shader? The first few comments in the article mention this.

[Edit: Another thing, what's the license of the code he publishes? I hate that, online code with no license... drives me mad.]

Re: Tangent calculation error

Posted: Sun Nov 10, 2013 8:05 am
by Kojack
Here's ogre's tangent calculation:

Here's the in-shader algorithm:

In both cases, the tangent is displayed as abs(normalize(tangent)). So no pixel should ever be black.

The model didn't have smoothing groups on in max (it was actually made for a vray render, I think vray's materials dealt with that separately, because it renders smooth), that's why the tangents are faceted around the neck and forehead.