resource codecs

[cyrfer]

Hi,
Does Ogre already have a framework to allow users to extend the supported file formats? If not, then I think a little code could go a long way. I'm not arguing to support different mesh formats in the engine, please don't misunderstand.

I would like to eliminate the off-line step of converting files into supported formats by allowing clients to add custom resource plugins, which would load the file into an existing resource type. For example, I could make an Entity with "my_model.obj" rather than "my_model.mesh", and the MeshManager would find a (manual?) resource loader that deals with .obj files. This loader would simply fill out a Mesh resource based on the .obj file, and then this Mesh resource would be available for use exactly as if it came from a .mesh file. This allows me to use the Entity class without changing any of the high level code, only the file name was different and no off-line conversion was necessary.

I've tried searching on keywords related to this topic, so forgive me if it has been beaten to death already. I did find someone that wrote something to load OpenFlight files, but it seems less elegant because the ManualObject was needed, rather than the (recommended?) Entity class. One of the reasons I chose to use Ogre is because I thought this abstraction might exist in some form, if not now, then eventually, because Ogre has other helpful abstractions.

Thanks!

[Frenetic]

Does Ogre already have a framework to allow users to extend the supported file formats?

Yes. Ogre is just about as extensible as it is possible to be!

Have you tried looking at the API docs? The Codec and Serializer classes might be a good place to start.

You can also look at the Ogre::Root class ctor to see how Ogre sets up its own built-in codecs, serializers, factories, etc.

Also note that the specialized SceneManagers, RenderSystems, etc are already plugins themselves! So look at those to see how to write your own plugin. Basically Root calls functions in the plugin when it loads and inits, and those functions allow you to register new codecs, factories, managers, serializers, etc with Root.

but it seems less elegant because the ManualObject was needed, rather than the (recommended?) Entity class

Mesh/SubMesh data is stored in vertex and index buffers. You can use ManualObjects to build vertex and indes buffers for you (and then you can easily create a Mesh resource out of the ManualObject's buffer) ... it's generally easier to do than operating on the buffers directly.

[cyrfer]

Does Ogre already have a framework to allow users to extend the supported file formats?

Yes. Ogre is just about as extensible as it is possible to be!

Have you tried looking at the API docs? The Codec and Serializer classes might be a good place to start.

You can also look at the Ogre::Root class ctor to see how Ogre sets up its own built-in codecs, serializers, factories, etc.

Also note that the specialized SceneManagers, RenderSystems, etc are already plugins themselves! So look at those to see how to write your own plugin. Basically Root calls functions in the plugin when it loads and inits, and those functions allow you to register new codecs, factories, managers, serializers, etc with Root.

@Fenetic: Thanks for the reply. I didn't make it clear from this question, but I am talking about Mesh resources. I am already familiar with the classes you mentioned. While I agree that Ogre can be extended for many things, it seems there is no existing built-in framework to do what I am talking about with the Mesh resource. From the classes you mentioned, I believe the answer to my question is 'no'. I know 'codecs' exist for Images, but nothing like that exists for reading a Mesh as far as I can tell.

I think the community would appreciate a framework to create Mesh resources from various file formats. The Ogre Mesh resource is a great class, and I don't want to change it, but I would like to be able to avoid an offline conversion step and create Meshes from various file formats, including encrypted formats.

Thanks.

[SunSailor]

Actually, it never was and most likely never will be recommended or wanted to use other mesh formats as the native one, as this one already is pre-optimized. There are a lot of exporters and converters available, so there doesn't seem to need for another interface for this, at least in my opinion.

[Frenetic]

No, it is quite easy to create Mesh resources, and there are lots of ways to handle custom formats at runtime.

ManualObject::convertToMesh is the easiest way to create a Mesh resource. Simple as that! It Just Works.

If you really want to do things the Hard Way just because and make your class a plugin that works more like the "built in" MeshSerializer-based set-up, while you can't explicitly override the use of MeshSerializer in Mesh::loadImpl, you can override the "Entity" MoveableObjectFactory using Root::addMovableObjectFactory with your own Factory, so you can just go createEntity("MyEntity", "anotherformat.obj") and your plugin can then load the resource like Ogre loads dot-mesh files. (You should use Root::getMovableObjectFactory beforehand so your custom factory can call the default one if the user isn't trying to load the custom format.) [edit] And don't forget to call MeshManager::createManual, etc.

It's all just a matter of knowing how Ogre works, and what code to write.

Also, if you're loading an ASCII file instead of a binary file, you might be better off using std::fstream to parse it instead of the built-in Serializer methods.

But SunSailor does have a point; using dot-mesh takes care of a lot of dirty work you really don't need to bother with, so you shouldn't try and load other formats at runtime unless you have a really good reason.

[cyrfer]

Actually, it never was and most likely never will be recommended or wanted to use other mesh formats as the native one, as this one already is pre-optimized. There are a lot of exporters and converters available, so there doesn't seem to need for another interface for this, at least in my opinion.

I really think you've missed my point. Forgive me for not knowing how to make it clearer. I can repeat my goals, but I don't know if that will prove I am talking about something completely different.

My goals are this:
- 1: allow my code to treat various file formats the same as the .mesh file format, resulting in the creation of the wonderful run time format known as Ogre::Mesh.

That's it. I don't want a different run time format, I am in love with the existing one. I also love being able to use that format in my code (via Entity), instead of using the ManualObject that people keep trying to pass off as an alternative.


A formalized framework to support my goal has some very useful benefits:
1 - allows the community to develop conversion tools from other formats so that Ogre::Mesh will rule them all. From what I've seen, only off-line conversion tools exist, and I have not seen the best results from them. I think their code would be improved if the community knew which one of the redundant projects to invest in. Also, off line tools are fine for some, but on-line tools make develop go faster. With on-line tools, you can still make off-line tools, but I don't think anyone would care when given option for the on-line support.

2 - a special case of benefit #1, but worthy of mentioning separately, is the fact that encrypted formats can now be used. Many of the hobbyists using Ogre probably don't care about encrypted formats, but for the businesses trying to see if Ogre is a good fit, support for encrypted formats is a requirement. Currently, I think Ogre could do more to support encrypted formats, and that is why I am bringing up this topic.

3 - as mentioned in benefit #1, the community will grow stronger because redundant develop of off-line tools will stop. The Ogre Add-ons projects could be a place to host the development of the on-line conversion tools, if only the main Ogre interface made it easier to develop on-line conversions.

Cheers.

[cyrfer]

No, it is quite easy to create Mesh resources, and there are lots of ways to handle custom formats at runtime.

ManualObject::convertToMesh is the easiest way to create a Mesh resource. Simple as that! It Just Works.

ManualObject is one way, but it avoids the whole point of the resources.cfg script. Does another way exist that benefits from Ogre's resource management?

If you really want to do things the Hard Way just because and make your class a plugin that works more like the "built in" MeshSerializer-based set-up, while you can't explicitly override the use of MeshSerializer in Mesh::loadImpl, you can override the "Entity" MoveableObjectFactory using Root::addMovableObjectFactory with your own Factory, so you can just go createEntity("MyEntity", "anotherformat.obj") and your plugin can then load the resource like Ogre loads dot-mesh files. (You should use Root::getMovableObjectFactory beforehand so your custom factory can call the default one if the user isn't trying to load the custom format.) [edit] And don't forget to call MeshManager::createManual, etc.

Great! That sounds like it might be the right place to start developing my support for encrypted formats. Thanks! Do you know if Ogre will try to declare these new resource formats when it parses the directories specified in the resources.cfg script?

My goal is not to do it the "hard way", but rather to keep things _easy_ in my code. Developing different code paths for different file formats (even though I just want Mesh resources) is really a PITA. I just want to make an Entity with my model - not a ManualObject (or something else) and then convert to a Mesh resource and then finally make an Entity...I'm sure you know what I mean. I'm now considering that ManualObject will be a redundant class if a proven framework for using Entity/Mesh with various file formats exists. I agree ManualObject is useful at times in Ogre's current design, but it appears out of place given that Mesh resources work well with the resource management system. I guess you can still use it to help your code when defining a procedural mesh, but I've never had trouble with just using the MeshManager API directly.

It's all just a matter of knowing how Ogre works, and what code to write.

I agree. I suspected some level of support for this already existed, but after trying to find the right development path for a while, I decided I would ask the experts.

Also, if you're loading an ASCII file instead of a binary file, you might be better off using std::fstream to parse it instead of the built-in Serializer methods.

Thanks for the tip, I'll look into it.

But SunSailor does have a point; using dot-mesh takes care of a lot of dirty work you really don't need to bother with, so you shouldn't try and load other formats at runtime unless you have a really good reason.

This sounds like another implementation tip. Can you elaborate about what dirty work you expect I'll need to do?

Thanks for the discussion, it's paying off!

[haffax]

Do you know if Ogre will try to declare these new resource formats when it parses the directories specified in the resources.cfg script?

Ogre doesn't even know what this resource.cfg thingy is. Ogre doesn't parse it, the example framework does, just look at the source it is only three or four lines for the parsing. Also the resource.cfg doesn't store resource types, only pathes to archives and archive types. So there is no way to derive any types from it. But when you parse it yourself you can add whatever info you need. You have to register the movable object factory yourself and handle types in your factory.

[Frenetic]

Note haffax's post.

Also the resource.cfg doesn't store resource types, only pathes to archives and archive types.

All the Example framework does with resource.cfg, IIRC, is use it to call Root/ResourceGroupManager::addResourceLocation.

That sounds like it might be the right place to start developing my support for encrypted formats.

If you want to implement encrypted resources, you should do that at the Archive level, not on the level of individual resource types.

I just want to make an Entity with my model

SceneManager::createEntity doesn't take a filename, but the string name of a Mesh resource. Any Mesh resource will work.

You most likely don't need to use an custom EntityFactory, now that I think of it. You can probably just use a ResourceGroupListener that adds manual Mesh resources to the MeshManager for the file format(s) it recognizes. Or, after the fact, you can iterate over an existing list of files in the ResourceGroupManager. I see lots of options you can try.

I'm now considering that ManualObject will be a redundant class if a proven framework for using Entity/Mesh with various file formats exists.

Loading vertex and index data into a Mesh resource can generally be done one of two ways: By accessing the buffer directly (like this) or by using the ManualObject as a convenient wrapper around a vertex/index buffer you can then put in a Mesh resource with 1 line of code. Once you're done with the ManualObject, you can just delete it or let it go out of scope, and then it's gone and all you have is a Mesh buffer that for most purposes looks exactly the same as if it was loaded from dot-mesh.

In other words, ManualObject is the proven framework you're looking for, unless you just want to get down and dirty with the vertex buffer API. Admittedly, using raw buffers does give you a little more control. But another framework for working with vertex and index buffers would, most likely, just replicate what the ManualObject already does.

Can you elaborate about what dirty work you expect I'll need to do?

Dot-mesh takes care of all sorts of stuff that gets taken for granted. Pretty much any metadata about the mesh you would otherwise need to generate at runtime, the dot-mesh format stores. For instance, it stores LoD data, and edges for stencil shadows, and tangent vectors for normal-mapping, and stuff that exporters can add like custom bounding volumes that take into account animations. I'm not actually sure what it does internally, but sinbad often emphasizes that the dot-mesh format provides more than just a way to store a chunk of geometry.