Tangent calculation error
Posted: Fri Nov 08, 2013 11:59 am
I've been playing around with getting a high poly (around half a million tris) model from max into ogre (using easy ogre exporter). Everything is fine, it looks good.
That is until I tried applying a normal map and lighting shader. After lots of struggling (including several hours of debugging shaders before realising the hlsl compiler had optimised one of my texture samplers away, causing the specular map to shift into the normal map sampler), I've found that some of the tangents are being calculated as 0,0,0. These are valid triangles. Their positions and uv coords are fine, no degenerates or anything. This happens not only in Easy Ogre Exporter but also ogre's own Mesh::buildTangentVectors(), which I'm guessing EOE uses.
The cause seems to be triangles with small uv coord changes. The code that calculates the tangent and binormal vectors scales them by the area of the triangle. These scaled vectors are accumulated on a vertex to find the final tangent and binormal. But then it checks if either vector is zero length (using Vector3::isZeroLength), any zero length vectors are skipped, leaving the vertex with a blank tangent.
The problem is that the uv coords on the bad triangles have a delta of around 0.001 or smaller. This means the area comes out as under 1e-6. The isZeroLength method checks for zero length by testing if the vector's squared length is less than 1e-12. Since the tangent and binormal were scaled by the area (less than 1e-6) that means the squared length is less than 1e-12.
The uvs may sound too small, but this model is using 4096x4096 textures. On them, each pixel has a uv width of 2.4e-4. Even a 1024x1024 textures have a pixel width just below 1e-3. It's easy to have small triangles on such a high res texture, but ogre will reject them and break the tangents.
Is the area really supposed to scale the tangents? That seems a bit odd. I can imagine scaling by the angle between the vectors, but area seems wrong.
I found an article (while looking for alternative tangent code) that's very interesting: http://www.thetenthplanet.de/archives/1180
It shows how to do normal mapping with no tangents in the vertices. It doesn't even use tangents and binormals, it uses cotangents and cobinormals instead, calculated on the fly in the pixel shader. Apparently it can handle non orthogonal tangent spaces, which the regular code can't. I've converted it to hlsl and it seems to work well, but I don't know the performance hit.
That is until I tried applying a normal map and lighting shader. After lots of struggling (including several hours of debugging shaders before realising the hlsl compiler had optimised one of my texture samplers away, causing the specular map to shift into the normal map sampler), I've found that some of the tangents are being calculated as 0,0,0. These are valid triangles. Their positions and uv coords are fine, no degenerates or anything. This happens not only in Easy Ogre Exporter but also ogre's own Mesh::buildTangentVectors(), which I'm guessing EOE uses.
The cause seems to be triangles with small uv coord changes. The code that calculates the tangent and binormal vectors scales them by the area of the triangle. These scaled vectors are accumulated on a vertex to find the final tangent and binormal. But then it checks if either vector is zero length (using Vector3::isZeroLength), any zero length vectors are skipped, leaving the vertex with a blank tangent.
The problem is that the uv coords on the bad triangles have a delta of around 0.001 or smaller. This means the area comes out as under 1e-6. The isZeroLength method checks for zero length by testing if the vector's squared length is less than 1e-12. Since the tangent and binormal were scaled by the area (less than 1e-6) that means the squared length is less than 1e-12.
The uvs may sound too small, but this model is using 4096x4096 textures. On them, each pixel has a uv width of 2.4e-4. Even a 1024x1024 textures have a pixel width just below 1e-3. It's easy to have small triangles on such a high res texture, but ogre will reject them and break the tangents.
Is the area really supposed to scale the tangents? That seems a bit odd. I can imagine scaling by the angle between the vectors, but area seems wrong.
I found an article (while looking for alternative tangent code) that's very interesting: http://www.thetenthplanet.de/archives/1180
It shows how to do normal mapping with no tangents in the vertices. It doesn't even use tangents and binormals, it uses cotangents and cobinormals instead, calculated on the fly in the pixel shader. Apparently it can handle non orthogonal tangent spaces, which the regular code can't. I've converted it to hlsl and it seems to work well, but I don't know the performance hit.