mdl2mesh - Neverwinter Nights model converter

[Borundin]

I dont know if this is the appropriate forum but I did include a few screenshots that shows the result from this tool. The script itself is not much to show

Background
A couple of years back I discovered the NWN mod Community and NWVault in particular. I was thrilled! Before that it had been a struggle just to get hold of a simple tree with no textures to download and use in your project. Now you had all kinds of models for testing purpose: trees, tiles, boats, monsters, weapons, shields etc all available for instant download. Unfortunately the only way for me to get those into OGRE mesh format was to use one of the two Milkshape MDL importers and then the Ogre exporter. Even worse, both of the importers had several drawbacks and some bugs that were almost hopeless to get around.

So last summer I began coding a LUA script that would read those MDL files and output OGRE XML. This was an intresting project and I managed to convert a few models using it. However on every new model I tried some new unforseen bug or limitation turned up. Got discouraged and dropped the project in August. This summer I found new energy to take this on again and now the script is ready for release


Description
This script starts by asking for a few input parameters. Only the first which is the name of the model should be changed unless you really want to. It then parses the file while building the mesh and skeleton XML files and eventually the material file. Just before the end it also starts up the OGREXmlConverter in order to compile the binary mesh and skeleton files. You need to change the path inside the script (near top) so that it matches your configuration or else it wont find the program. There are a lot of debug text while running the script but nothing to worry about. I might change it to be more silent in later releases.

Another thing I should mention is that most MDL models use a skeleton even if it doesnt have too. For example static geometry such as trees, doors etc usually come with a skeleton. In OGRE I believe a skeleton will slow down the rendering of such objects. I might change the script later so that if it cant find any animations it wont create the skeleton link at all.

Im not a lawyer so I dont know exactly in what ways you can use these models in your own projects. Im pretty sure that you cant use the Bioware models at all except maybe for private testing purposes if you own a copy of NeverWinter Nights. But I believe that some of the content from the Mod community can be used even in commerical products if you get permission from the author. Bioware cant claim intellectual property unless it is obvious that the model is derived or based upon one of the Bioware orginal models.


Features

• Geometry
  Texture coords
  Normal generation (MDL format doesnt seem to include normals?)
  Bones
  Animations
  Basic material

Limitations

• Treats "danglymesh" as standard trimesh
  No emitters
  No events
  Only a few material properties supported yet (Transparency is on the TODO list)

Known bugs
Rotation of the bind pose bones. One example is the tail of the pig in the Swine hakpak that floats behind the body. Not very pretty =(


Downloads
The Zip contains mdl2mesh.lua, Lua executeable, a command file and a sample MDL file with its associated textures. Just unzip to a directory and double click the runme.bat file to get started.

mdl2mesh ver 0.9

And another Zip that contains an oak, an Uruk-Hai with one walk animation and a turtle with alot of animations. All converted to OGRE format (oak material fil was handedited to support transparency).

ExConverted


Media
Screenshots:

From Classic Dungeon 0.8.1

From Dwarven Halls

From LUSH: Version One


A movie with a goblin walking and slashing.

] goblin.avi (divx) From Cave Goblins (Full Pack)


Support
I will do what I can to provide support for this script. Sinbad has even hinted at this "project" ending up in the OgreAddons . Anyway if you do want to report a bug please check first that the model does show up properly in the MDLViewer from Bioware (it probably will, cause that app is very forgiving) and that it is not one of the known bugs or a limitation. If not then post in this thread a short description of the error and provide a link to where one could get the model (name of hakpak, model) and I will try to fix the problem.

Enjoy!

[CombatWombat]

Very nice! Thanks for your work on this Borundin

[Kentamanos]

Nice work Borundin!

[jacmoe]

Very professional! - very nice!

[skullfire]

Awesome1! Cant wait to get my hands on some of NWN media to test!! XD great work!

[spoot]

Looks sweet! Can't wait to try it out.

[Marc]

Wow that's really good timing, I thought a few weaks that converting nwn models to ogre for having testmodels for ones code is a great idea. Mine isn't ready to use. I'll just use yours

A few internal questions: Does your converte support binary models? How do you handle derived animations from parent models (setsupermodel entry) ?

[Borundin]

A few internal questions: Does your converte support binary models?

No just ASCII models. If they are binary I just convert them to ASCII with the NWN_compDcomp utility which btw handles batch conversions also.

How do you handle derived animations from parent models (setsupermodel entry) ?

mdl2mesh reads information from just one file so only the animations inside that file will get exported. So far I just cut n paste the animations I need from the supermodel into the MDL file if they arent there already. This could probably be solved in a better way so please post if you have any ideas

[Marc]

mdl2mesh reads information from just one file so only the animations inside that file will get exported. So far I just cut n paste the animations I need from the supermodel into the MDL file if they arent there already. This could probably be solved in a better way so please post if you have any ideas

No, I didn't get much further than the planing phase. My plan on this was, like yours, to not support supermodels for the beginning.

[Marc]

Rehi !

I tried your converter now. Quite cool, good work.

I didn't find a buggy model until now. I had to set those bits from the beginning to 0,0 for animated models though but that's probably no bug.

A minor feature wish: I'ld like to have the options as cmdline parameters so I can use it for batch processing. That'ld also testing on a lot of modells ...

The only thing I think about is optimality for performance in ogre. Perhaps some Ogre guru can tell us how big the impact on this is: You already mentioned the unneccessary skeleton for static models. I have some related point: It seems that atm your script generates one Ogre submesh for each nvn-mdl-bone. The number of different textures per nvn-mdl is much lower than the number of nvn-mdl-bones. As far as I understand, Ogre creates one gpu command/batch per submesh. But if multiple submeshes share the same material, it'ld be faster to merge them into one submesh. So i think grouping the submeshes generated from your script by textures and only use the nvn-skeleton for assigning the vertices to the ogre-skeleton could increase rendering performance of the converted models quite a lot on newer hardware. E.g. the ant-model I have open atm has 32 submeshes but only 4 materials.

[Borundin]

I didn't find a buggy model until now. I had to set those bits from the beginning to 0,0 for animated models though but that's probably no bug.

Well as you noticed the script do handle this but only if providing the correct parameters. It would be much more elegant if the script itself understood how to process the file. MDLViewer seems to do it so I know it is possible. Just havent found out how yet

A minor feature wish: I'ld like to have the options as cmdline parameters so I can use it for batch processing. That'ld also testing on a lot of modells ...

Absolutely, I have been thinking about this too. This will get into next version.

It seems that atm your script generates one Ogre submesh for each nvn-mdl-bone. The number of different textures per nvn-mdl is much lower than the number of nvn-mdl-bones. As far as I understand, Ogre creates one gpu command/batch per submesh. But if multiple submeshes share the same material, it'ld be faster to merge them into one submesh. So i think grouping the submeshes generated from your script by textures and only use the nvn-skeleton for assigning the vertices to the ogre-skeleton could increase rendering performance of the converted models quite a lot on newer hardware. E.g. the ant-model I have open atm has 32 submeshes but only 4 materials.

Oooops, didnt think about that but youre probably right. Changing this will have a big impact on the internals of the script unfortunately. Not sure I can fix this in the nearest future. Would be intresting to do it manually on a file first to see exactly what kind of performance gain you can get. Maybe create 50 ant models (in Ogre app) with and without submesh feature and see what fps you get?

[Borundin]

This script and media can now be downloaded from Ogreaddons/mdl2mesh. New versions will be available here in the near future.

[Marc]

Oooops, didnt think about that but youre probably right. Changing this will have a big impact on the internals of the script unfortunately. Not sure I can fix this in the nearest future. Would be intresting to do it manually on a file first to see exactly what kind of performance gain you can get. Maybe create 50 ant models (in Ogre app) with and without submesh feature and see what fps you get?

I'ld suggest not to change your script completly for doing that. First of all it seems to work very well the way it works now and second, that's a problem that could be handled with an ogre->ogre optimizer tool and might be usefull for other non-optimizing converters for other data too. I wrote such a script in php that does this but atm does not handle skeletons and animations. It only merges submeshes down. I'll try to remove the animation from the ant model and render it 100 times unoptimized vs. optimized by my script.

[Marc]

I've just seen that you put positions, normals and texcoords of vertices in different vertexbuffers. Is that good or bad with respect to speed or something else?

[Borundin]

I've just seen that you put positions and normals of vertices in different vertexbuffers. Is that good or bad with respect to speed or something else?

Actually it doesnt matter since the OgreXMLConverter will put them into the same vertexbuffer anyway. You can verify this by converting the resulting mesh file back to XML. It will look something like this:

Code: Select all

            <geometry vertexcount="251">
                <vertexbuffer positions="true" normals="true">
                    <vertex>
                        <position x="8.18252" y="-4.7993" z="78.7147" />
                        <normal x="0.320425" y="-0.896634" z="-0.305574" />
                    </vertex>
                    <vertex>
                        <position x="-0.841135" y="3.1201" z="85.2697" />
                        <normal x="-0.144225" y="-0.503157" z="0.852075" />
                    </vertex>

[Marc]

Ah good to know

I created 3 ogre models from the nwn-ant-mdl:

1. the model that is produced by your script
2. a model that is produced by taking the .mesh.xml created by your script and merged the vertexbuffers.
3. a model that is produced by taking the .mesh.xml created by your script and merged vertexbuffers and merged submeshes.

Rendering 100x the first one from a fixed camera position and resolution etc. gives 4fps. 2 gives mit 16fps and 3 gives me 80fps. 800x600, window, GForce 4 TI 4200

So merging the submeshes seems to be a good idea. What makes me wonder is why 1 and 2 have different framerates. One explanaition could be that model 1 includes the boneassignment entries while 2 does not. If that's the reason, your idea of removing unneccessary skeletons from static models is a good idea too. Otherwise I have no explanaition for this. I didn't though that just having a skeleton has such a big impact on speed (I don't animate 1. I just load and show it like 2 and 3)

I could mail you the optimized models if you'ld like to test for yourself.

[Borundin]

Yes, removing unneccesary skeleton makes a huuuge difference in my tests at least. Doesnt matter if you actually start an animation as far as I know because it still adds alot of overhead (software).

Anyway you have some intresting results. I have to analyze this and see what can be done about it. Seems pretty clear that merging submeshes with the same material gives a big performance gain and that alone makes it a top priority for me

[Marc]

and that alone makes it a top priority for me

Good enough for me. I'm waiting for the next version then

[Borundin]

Just a short update...

I finally had time to do some tests on my own with submesh grouping. I rewrote the script to group submeshes with the same material and then did a test with 50 walking UrukHais. The old method with one submesh per bone gave 28 FPS in my test setup. New version gave 58 FPS which is alot better. Not the same performance gain as Marc got in his tests but still good.

I hope to have a new version of mdl2mesh ready this weekend. Command line options should also be in at that time.

[skullfire]

Just a short update...

I finally had time to do some tests on my own with submesh grouping. I rewrote the script to group submeshes with the same material and then did a test with 50 walking UrukHais. The old method with one submesh per bone gave 28 FPS in my test setup. New version gave 58 FPS which is alot better. Not the same performance gain as Marc got in his tests but still good.

I hope to have a new version of mdl2mesh ready this weekend. Command line options should also be in at that time.

awesome work dude. Cant wait for the next version. i however im a little concerned with some things.. like for instance, all the models on the demo seem to be facing the -y direction... including the tree which actually makes me have to rotate it( pitch(Degree(-90)) ). Then.. when its up(teehee), the mesh is higher when its walking than when its standing on its ground.. is this part of a problem because I pitched it?

[Borundin]

awesome work dude. Cant wait for the next version.

Thanks

i however im a little concerned with some things.. like for instance, all the models on the demo seem to be facing the -y direction... including the tree which actually makes me have to rotate it( pitch(Degree(-90)) ).

Yes this is a problem. I have actually got so used to it that I forgot to mention it in the first post. Would like to help you here but this is rather low on my priority list for two reasons. 1) This is easily solved i OGRE by rotating the scenenode you attach it to. 2) I have to put quite a lot of math functions inside the script just to fix this and I really think OGRE is better suited for this than a Lua script (yes this is another way of saying that I dont have the skill needed )

Then.. when its up(teehee), the mesh is higher when its walking than when its standing on its ground.. is this part of a problem because I pitched it?

I have seen this too. Could be the pitching or maybe the bind pose is positioned "lower" than where the skeleton puts it during animation. Most animated MDL models have a pause animation. Maybe you could use this instead of no animation at all when just standing on the ground?

[Marc]

Then.. when its up(teehee), the mesh is higher when its walking than when its standing on its ground.. is this part of a problem because I pitched it?

I'm not sure if this is related to your problem. But I experienced that some nwn-models look like being in a strange pose if you don't apply an animation to them. The animation then arranges the skeleton in a way how it should be.

[skullfire]

Hmm.. ill try the the pause animation.. that could do the trick. thanks dude!

[Borundin]

New version of mdl2mesh.lua have been uploaded to OgreAddons. Will probably take a few hours before it trickles down to anonymous access but if you wish you can also download the latest version from this link

Changes:
- Grouping of submeshes with same material (performance)
- Alpha rejection in materials
- Command line arguments
- possible to skip skeleton creation (useful for static geometry)

The grouping of submeshes seems to have solved other issues with body parts floating behind main body (G_Ant model for example) and with this version I cant seem to find any model that needs bone rotation disabled.

Command line arguments is available in this version. If you dont provide any arguments at all the old interactive mode is enabled.
Syntax:

lua.exe mdl2mesh.lua [-s <scale factor>] [-noskel] [-nobonerot] [-noanimpos] [filename]

[Marc]

Wow, that was a lot faster than I expected. This is really cool. Thanks a lot.