[GSoC 2013 - accepted] Improve Progressive Mesh

[Pforce]

Hello everyone. Just have couple of questions regards to this GSoC project. I know this is still work in progress and all that.

OgreXmlConverter

Are there any plans to fix OgreXMLConverter tool during this project? I see that you have done most code into the OgreProgressiveMeshGenerator class, but for certain things you have also dipped into the tool side a bit. I know this project is about the progressive mesh in OgreMain but would it be possible to take the converter tool to the next level and make it, first of all work for LOD generation, and make these new features options in there (like the new compression)?

The progressive mesh class looks great for runtime things like you have showed with the refreshed sample, but it would be great to make the tool also available. I've built your branch and the xml converter is broken, I've tracked the bug to a line where it clones the input mesh for the LOD generation, so once the original input mesh gets converted the LOD levels are not there. Basically to me it seems this tool has been completely broken for about 7 months now, am I getting the correct impression here? Line and commit im talking about: https://bitbucket.org/sajty/ogre/commit ... in.cppF670 This clone should be removed to make it actually export the LODs.

Also once I fix that the tool is still quite broken, for example the -p option does nothing, it creates one single LOD level with exactly the same face count as the original mesh. Using the -p and -s Distance seem to have no effect either. I've not yet been able to export any more than one LOD level (after my fix) with the tool, no matter what i give to -l option.

Is there a technical reason why LOD levels cant be generated to a input binary .mesh and exported into .mesh.xml? I see the docs say this but looking at it, isnt it the exact same codepath? Import the file to a MeshPtr and feed that into the LOD code and export the result as xml or binary?

As said, I know this is work in progress and it is perfectly fine the tool/code there might not be in working state all the time. I just wanted to ask what are the plans on the OgreXMLConverter, if improving it is in scope for this project.

Other stuff

With your new progressive mesh code do you think its possible to make a tool (or make OgreXmlConverter support it) that would reduce the poly count of the Mesh in place, not generating LOD levels but taking in the submeshes and reducing its complexity and exporting that version to a file. I've tried to play around with the current code to get a x LOD level out to export it as its separate mesh, but I haven't really found a way to do this. Are there technical problems why this could not be supported? I know there are Blender and whatnot tools that you can just click decimate on, but a cmd line tool to reduce mesh complexity could be very useful for offline tooling. I would be interested in implementing this kind of tool, so its it possible? I know it might not be in the interests of Ogre in general to make this kind of tool, as its a game engine and artists can do this in their modelling tools.

Thanks for working on this!

P.S. Remember to remove those nullptr:s

Edit: Just to clarify, I've tested the earlier SDK shipped versions of the xml tool and they are similarly broken, most LOD options do nothing or wont do whats documented in -h. I'm no saying you broke the tool

[sajty]

Hi, Sorry for not answering earlier. I was on holiday for the weekend.
The refactoring took longer than expected. Soon I will create diagrams of the design.

Are there any plans to fix OgreXMLConverter tool during this project? I see that you have done most code into the OgreProgressiveMeshGenerator class, but for certain things you have also dipped into the tool side a bit. I know this project is about the progressive mesh in OgreMain but would it be possible to take the converter tool to the next level and make it, first of all work for LOD generation, and make these new features options in there (like the new compression)?

I have mostly used the binary format and XML is lagging behind. The refactoring and other stuff was higher priority for me then tools and XML, but I will look into your issue soon.

The progressive mesh class looks great for runtime things like you have showed with the refreshed sample, but it would be great to make the tool also available. I've built your branch and the xml converter is broken, I've tracked the bug to a line where it clones the input mesh for the LOD generation, so once the original input mesh gets converted the LOD levels are not there. Basically to me it seems this tool has been completely broken for about 7 months now, am I getting the correct impression here? Line and commit im talking about: https://bitbucket.org/sajty/ogre/commit ... in.cppF670 This clone should be removed to make it actually export the LODs.

Yes, that cloning should not happen. Sorry, I don't have noticed it, because I was just trying to convert as a test, but don't have checked that the output is correct. Thanks for reporting.

Also once I fix that the tool is still quite broken, for example the -p option does nothing, it creates one single LOD level with exactly the same face count as the original mesh. Using the -p and -s Distance seem to have no effect either. I've not yet been able to export any more than one LOD level (after my fix) with the tool, no matter what i give to -l option.

Is there a technical reason why LOD levels cant be generated to a input binary .mesh and exported into .mesh.xml? I see the docs say this but looking at it, isnt it the exact same codepath? Import the file to a MeshPtr and feed that into the LOD code and export the result as xml or binary?

As said, I know this is work in progress and it is perfectly fine the tool/code there might not be in working state all the time. I just wanted to ask what are the plans on the OgreXMLConverter, if improving it is in scope for this project.

Yes, It's in the scope, but as said still lagging behind.

With your new progressive mesh code do you think its possible to make a tool (or make OgreXmlConverter support it) that would reduce the poly count of the Mesh in place, not generating LOD levels but taking in the submeshes and reducing its complexity and exporting that version to a file. I've tried to play around with the current code to get a x LOD level out to export it as its separate mesh, but I haven't really found a way to do this. Are there technical problems why this could not be supported? I know there are Blender and whatnot tools that you can just click decimate on, but a cmd line tool to reduce mesh complexity could be very useful for offline tooling. I would be interested in implementing this kind of tool, so its it possible? I know it might not be in the interests of Ogre in general to make this kind of tool, as its a game engine and artists can do this in their modelling tools.

Yes, I thought about that idea too. However this is also related to tools. It may going to be just a separated tool (without profiling support). If you want to use blender to export Lod levels, you need to use manual Lod levels, because blender is changing vertex positions, which requires separated vertex position buffers. This makes bigger memory usage, than the ogre built-in reduction algorithm. So there are technical limitations. To do this (universally in any editor/exporter), just export the mesh with different Lod levels separately, then in the LodConfig in the original mesh for each level set manualMeshName to the mesh to show. The exporter could automate this for you. Maybe ask the author of the exporter for this feature.


Overall you are completely right, but I don't get to those yet. Please be patient. However your post will help giving more priority to it, because nobody complained about it until now.

[sajty]

Hi, the refactoring still needs some final touches, but I would like to describe the improvements here. First of all, it took much more work then expected, but in the end, it was getting much cleaner then I have expected too.

new CMake flags:
-OGRE_CONFIG_ENABLE_MESHLOD: Removes everything from OgreMain related to MeshLod. It will give a small speedup if you don't use MeshLod. MeshSerializer will skip the Lod part in the .mesh file.
-OGRE_BUILD_COMPONENT_MESHLODGENERATOR: It will enable Lod generation at runtime. (OGRE_CONFIG_ENABLE_MESHLOD needs to be enabled)

I have splitted the algorithm into 6 components:
1. LodData: The core of the algorithm, containing the Mesh network representation.
2. LodCollapseCost: Provides an interface for collapse cost calculation algorithms (like curvature, quadric error, profiling, outside weight, normals)
3. LodInputProvider: Initializes LodData based on a source (It can be mesh or buffered(for threading)).
4. LodOutputProvider: Bakes the Lod to a target (It can be mesh, buffered, mesh+compressed and buffered+compressed)
5. LodCollapser: This will reduce the smallest costing vertices to the requested amount of vertices. Creates/destroys faces.
6. MeshLodGenerator: Frontend, which can receive LodConfig. It will create/run the required internal components to generate the requested LodConfig. Only this class is using the LodConfig.

Other notes:
-Threading/non-threading option is moved to the LodConfig too.
-LodCollapseCost can be chained. For example if you use profiling and outside weighting, then: LodCollapseCostProfiler ==> LodCollapseCostOutside ==> LodCollapseCostCurvature is chained by MeshLodGenerator.

[sajty]

I've checked this topic and felt it is a bit unended, so I decided to add a final post here.

The changes were merged with upstream long time ago to v1.10 branch.
Changes which weren't mentioned here:
-Refactoring is fully complete.
-I have created bunch of unit tests including mesh serializer tests.
-Fixed XML format and tools. Added new flags to the tools for new lod features like -autogen flag.
-Added more documentation and a wiki page: http://www.ogre3d.org/tikiwiki/MeshLod

Also I would like to thank my mentor wolfmanfx for the support!
I really enjoyed working on this task.

[dark_sylinc]

I saw that you've removed a lot of functionality from XMLConverter and left it in MeshUpgrader (like tangent generation, edge list generation, etc).
I almost shit myself.

This is a TERRIBLE idea. I'm in for refactoring duplicated code, but the functionality must be available in both tools.
The common export process in a pipeline is Exporter -> Generate XML file -> Convert to Mesh.
In this process, xmlconverter generates the tangents, edge lists, etc for you which is very, very convenient. It also breaks existing exporters.

Murat suggested chaining xmlconverter with meshupgrader. I dislike this idea. An export pipeline is already hard and painful to setup. Adding an additional node in the chain means another possible point of failure.
Furthermore, this slowdowns the export process when the mesh in question is big (>10MBs). Having fast iteration times is a must.
And besides this means the export tool needs to be updated.

Updating export is a no go. Some tools are still usable but not maintained, while others are hard to recompile:
If you have to compile (e.g.) for Maya 2010, 2011 & 2012; you need to prepare the SDKs of each, and use the right CRT libs that match that of Maya's. It's a painful process that one prefers to avoid.
Furthermore, this means that you then have to update the exporter in each of the artist's PCs. If you have 10 and 2 of them are having trouble installing the update, it makes everything harder.
AAA Studios have the luxury of having all their artists use the same modelling tool version, but this often isn't the case for everyone else.

Long story short, updating an exporter isn't easy. Don't expect them to adapt to interface breaking changes.

My suggestion is to use a common CPP file in both tools, like this:

Code: Select all

void printCommonCommandLineHelp()
{
/* ... */ 
}

void doCommonParameters( argc, argv )
{
/* ... */ 
}

void finalizeCommonParameters( argc, argv )
{
/* ... */ 
}

Murat suggested instead to merge both XMLConverter and MeshUpgrader into one. If one learns from Linux, is that God-Tools should be avoided, as they end up becoming harder to maintain.
Cmd Tools should to one function, or two at top.

We (Murat & I) both agree though that it could be possible to merge into one tool through refactoring, and Murat says that my idea just replaces a god-tool with a god-file.
Unless someone offers to do the refactoring, something needs to be done to fix the XMLConverter. In it's current state it's unusable and a shot in the foot.

If no one fixes it soon, I will. But it's not gonna be pretty.

[sajty]

The problem was that they evolved differently like a separated fork. XMLConverter had some bugs, while MeshUpgrader had different ones. Even if you looked at the implementation for the same functionality, it was different.

So in the past it was definitely not maintainable in its form and I needed to find an optimal solution:
-XMLConverter is supposed to have only XML==>Mesh and Mesh==>XML functionality (while generating Lod levels has nothing to do with it).
-If you want to upgrade your mesh with edgelists, lod levels, tangents(if you don't export them), then you should use MeshUpgrader.
This seems to be a clean solution.

If we would add these back to XMLConverter, then XML converter can do nearly everything, except converting to older mesh formats. Now you may come: "But in the exporter they should have the possibility to export to older mesh formats! Lets patch it!" Which ends up adding all functionality to XMLConverter...

Furthermore, this slowdowns the export process when the mesh in question is big (>10MBs).

You mean loading the file twice from HDD? On the second run, it should already be cached in RAM by the OS and it would only differ in few milisecs in processing time. Even if you want to batch convert (and generate lods) for 1000 meshes, the difference will be few second in a few minutes processing time.

Your suggestion is to create a (half) god tool, which you explained why it is wrong.

Lets vote: single god tool or separate XML converting/mesh upgrading.
Other ways doesn't sound to be optimal solution for me. The only reason for suboptimal solutions is backwards compatibility, which is also coming into game here. Maybe I was a bit naive to assume that the exporters will adapt it, but to improve something you need to do breaking changes sometimes...

Edit: The artists in AAA studios doesn't have to upgrade Ogre and/or tools, because Ogre can load older mesh versions. I would even recommend using the Ogre SDK version which the exporter is designed for. They can upgrade the mesh before a release if they want with MeshUpgrader.

[dark_sylinc]

The problem was that they evolved differently like a separated fork. XMLConverter had some bugs, while MeshUpgrader had different ones. Even if you looked at the implementation for the same functionality, it was different.

I won't argue that.

So in the past it was definitely not maintainable in its form and I needed to find an optimal solution:
-XMLConverter is supposed to have only XML==>Mesh and Mesh==>XML functionality (while generating Lod levels has nothing to do with it).

Well, that's the thing. During the XML ==> Mesh thing, many exporters or tools don't include tangent generation (VERY important) or edge list (somewhat important) or not even Lods.
XMLConverter sits in a key place in the pipeline; and thus its ability to do that (plus autogenerate Lod) is very, very important and frequently used.

MeshUpgrader on the other hand, is used sporadically. Often just to silent the Ogre.log warning that the mesh format is out of date (although there are other reasons)

In fact, the MeshUpgrader's code has traces of proof that its code has been backported from XMLConverter, this in MeshUpgrader's code:

std::cout << "\nXML already contains level-of detail information.\n"
"Do you want to: (u)se it, (r)eplace it, or (d)rop it? ";

-If you want to upgrade your mesh with edgelists, lod levels, tangents(if you don't export them), then you should use MeshUpgrader.
This seems to be a clean solution.

Unfortunately it's not clean, particularly since it breaks existing interfaces (may be if Ogre was a brand new engine, this discussion would be different).

Lets vote: single god tool or separate XML converting/mesh upgrading.

Actually I prefer separate tools but sharing a cpp file for the common parameters.

Code: Select all

//XMLConverter
struct XmlOptions
{
 /* ... */
 SharedParams sharedParams;
}

void help(void)
{
    /* ... xml print help code here ... */
    printSharedHelp();
}

XmlOptions parseArgs(int numArgs, char **args)
{
    XmlOptions opts;
    /* ... xml param parsing here ... */
    parseSharedArgs( numArgs, args, &opts.sharedParams );
}

void XMLToBinary(XmlOptions opts)
{
   /* ... xml -> mesh conversion here ... */
   processSharedCode( mesh, opts.sharedParams ); //Will perform Lod, tangents, edge list, etc.
}

//>>>>>      ------------      <<<<<<
//MeshUpgrader
//>>>>>      ------------      <<<<<<

//Note I've added two arguments.
void parseOpts(UnaryOptionList& unOpts, BinaryOptionList& binOpts, int numArgs, char **args )
{
    parseSharedArgs( numArgs, args, &opts.sharedParams );
}

int main( int numArgs, char **args )
{
/* ... upgrade code here ... */
   processSharedCode( mesh, opts.sharedParams ); //Will perform Lod, tangents, edge list, etc.
}

It's not a perfect solution, but solves the duplicate-code pattern while providing the functionality in both tools at the same time. And it's not that hard to do either.

You mean loading the file twice from HDD? On the second run, it should already be cached in RAM by the OS and it would only differ in few milisecs in processing time. Even if you want to batch convert (and generate lods) for 1000 meshes, the difference will be few second in a few minutes processing time.

Loading from HDD and populating all the buffers take time. Time it. Some meshes are 40 to 60MB easily.
Batch conversion is usually undesired because different assets need to be exported with different flags.

Edit: The artists in AAA studios doesn't have to upgrade Ogre and/or tools, because Ogre can load older mesh versions. I would even recommend using the Ogre SDK version which the exporter is designed for. They can upgrade the mesh before a release if they want with MeshUpgrader.

An AAA studio (or an Indie for what it's worth) would never risk to perform regular testing using old assets only to change a substantial amount of their assets even weeks before release just to "upgrade".

Cheers

[scrawl]

Ideally you put shared functions in a shared library, although that does add increase the complexity of the build system slightly. Static library would be a last resort, at least then you wouldn't compile the exact same source files twice.

[sajty]

It's not a perfect solution, but solves the duplicate-code pattern while providing the functionality in both tools at the same time. And it's not that hard to do either.

Thats the point: "It's not a perfect". You just shift the problem one layer down. Now you have duplicate machine code, because you have 2 executables with same functionality. You compile it twice too.

[dark_sylinc]

Thats the point: "It's not a perfect". You just shift the problem one layer down. Now you have duplicate machine code, because you have 2 executables with same functionality. You compile it twice too.

It's not perfect, but in this case I'll prefer compiling this twice than having to do this twice at run time.

[sajty]

So if I understand right, your main concern is calling 2 executables for importing and upgrading.
Then we should go with the god executable. It doesn't have duplication anywhere and it doesn't require layers, cross executable cpp files, etc.
90% of the functionality and flags is equal anyways, so I think going your way would increase complexity more than merging the executables.

[dark_sylinc]

I just realized that the MeshUpgrader's code is smaller than I thought. I think you're right. Merging the executables is a better idea.
One thing though, the final name should remain "OgreXMLConverter.exe" for backwards compatibility with existing tools.
Barely any other tool actually integrates the MeshUpgrader.

[scrawl]

Or you could give it a more appropriate name (say "OgreMeshTool") and create a symlink OgreXMLConverter to that for compatibility purposes.

[dark_sylinc]

Or you could give it a more appropriate name (say "OgreMeshTool") and create a symlink OgreXMLConverter to that for compatibility purposes.

If that's so, copy paste post build operation for Windows XP, symlink for +Vista (requires asking for admin privileges). Linux symlink.

[scrawl]

Eh, what? Symlinks need admin priviledges on windows? Nevermind, then...

[spacegaier]

I also like the idea of combining the two. Much more streamlined.

As a note to whomever is going to tackle that change: Throughout the Ogre code base there are a few mentions of the OgreMeshUpgrader tool. Those might need to be changed as well, perhaps to something like:

WARNING: XXX is an older format (0.5). You should upgrade as soon as possible using either the legacy OgreMeshUpgrader tool (Ogre version <= 1.9) or the new OgreXMLConverter (Ogre version >= 1.10).

[scrawl]

I believe there is no point in mentioning the legacy name of the tool in that message, because you'll only see the message when using an Ogre build >= 1.10, which means you'll want to use the new tool.

[spacegaier]

True from a technical point of view, however it might confuse people that are used to seeing the OgreMeshUpgrader mentioned there. But I am fine with it, either way as long as it is adjusted in principal.

[AshMcConnell]

@scrawl I am using your progressive mesh code in a 1.9 project. I have upgraded all the meshes in my race track using a OgreMeshUpgrader with a batch file using the autoconfig features. I'm just a bit confused as I don't see any improvement in the FPS and when I output framestats.triangleCount it is the same as with the non-LOD'd meshes. Am I doing something wrong?

(It could be that I'm CPU limited at the minute - hence the lack of FPS improvement with LOD, but I expected the triangleCount to be different).

Thanks for your help!

[Wolfmanfx]

@Ash
Maybe your problem are the shaders - you should try to use loded materials and decrease the shader work too

[AshMcConnell]

@Ash
Maybe your problem are the shaders - you should try to use loded materials and decrease the shader work too

I have very little shaders on the track (only the track ribbon itself). It's all fixed function at the moment. I do have LOD on the car materials - but I'm always in-car, so that shouldn't make a difference. I would like to check that the LOD is properly enabled for me and I haven't made a mistake

[scrawl]

Hi,

sajty worked on progressive mesh, not me But I'll try to answer anyway.

Whether you see an FPS improvement really depends on your graphics card and the total number of triangles, but as a rule of thumb on today's desktop cards anything below 1 million triangles should not be a problem at all and reducing that is probably not worth your time. Batches are usually the FPS killer (especially in Ogre 1.x).

when I output framestats.triangleCount it is the same as with the non-LOD'd meshes.

IIRC (could be wrong about this) the progressive mesh leaves the vertex buffer as it is, the reduction comes from changing the index buffer to "skip" over vertices, thus reducing the number of faces drawn.
I don't know how the triangleCount is calculated, but if it's something like (numVertices/3) that would explain why it doesn't change.

I would like to check that the LOD is properly enabled for me and I haven't made a mistake

If in doubt, use extreme lod values on one mesh then enable wireframe mode and see if it changes when you move away from it

[sajty]

I would recommend debugging in this order:
1. Check distances:
For the LOD level, which doesn't want to work for you, try to reduce by 100% without changing lod strategy or lod distance (distance depends on strategy). If the Object disappears for you in distance, then these are set up correctly. (common mistake is to select the wrong strategy)
2. Check the reduction:
You can force LOD level (ignoring distance and strategy) with mEntity->setMeshLodBias(1, lodID, lodID) http://www.ogre3d.org/docs/api/1.9/clas ... 5176916564
This way you can go nearer to the model and see how the mesh LOD level looks.


Autoconfigured LOD (and LOD in general) is useful on high poly models, but if you have a low poly model then the reduction amount will be minimal. It depends of how much artifact the "cheapest" reduction will produce.
Triangle count should always be smaller on a reduced mesh, because triangle count has nothing to do with vertex buffer, but with index buffer. So if you see the same triangle count in distance, than you should have some problems with the LOD.

[AshMcConnell]

sajty worked on progressive mesh, not me But I'll try to answer anyway.

Apologies guys, I have been following sajty's work for a long time, I'm not sure why I thought it was scrawl, notch it up to parental lack of sleep perhaps

Thanks guys, some good suggestions, I haven't had a lot of time to try them out as this weekend has been very busy. I tried forcing the LOD to the lowest detail and the FPS rose quite a lot although the batches also reduced quite a lot so that was probably the reason. I am getting a lot of artifacts, so I'll have to tune them individually (which is beneficial in any case).

I am using the default strategy at the moment - I am using 1.9 so I *think* that is distance, but the LOD in the new OgreMeshUpgrader uses PixelCount IIRC. I thought the strategy would be stored in the mesh?

I'll try changing the default strategy using: -

Code: Select all

Ogre::LodStrategyManager::getSingleton().setDefaultStrategy(Ogre::PixelCountLodStrategy::getSingletonPtr());

Thanks for your help, hopefully I'll get a chance to play more and tune it this week

All the best,
Ash

[sajty]

The 1.9 release has introduced new LOD strategies, which broke backwards compatibility, so you need to use AbsolutePixelCountLodStrategy instead of PixelCountLodStrategy. I have added a typedef in v1.10 to make it compatible with older Ogre versions, but it was not getting into v1.9. This causes even more trouble in the mesh file, because the strategy name differs between v1.8 and v1.9 and the mesh file stores the names only. However v1.10 will be able to accept v1.8 and v1.9 strategy names properly when loading a mesh. In v1.9, if the strategy name was not found, the default strategy is used.

The default strategy has always been the DistanceLodStrategy (DistanceLodBoxStrategy in v1.9), but you can change it with setDefaultStrategy manually.
In the MeshUpgrader you can select what kind of strategy you want to use.

DistanceLod*Strategy: faster to calculate distance; on higher screen resolution you may notice more artifacts then on low resolution; LOD is equally efficient on all resolutions.
*PixelCountLodStrategy: more calculation; artifacts are equal on all resolutions; on a 4k display the first LOD level will swap too far away making LOD inefficient.