Ogre and Unicode

[PsyCowboy]

Hi all,

---- Description of the issue ----

I have troubles with Ogre and pathes containing non-ASCII characters.
If pathes contains non-ASCII characters, Ogre simply cannot open them.

For example, Trying to open this file t0无 implies this exception:

Code: Select all

12:50:57: Added resource location '/home/fabien/Desktop/t0?/worlds/textures' of type 'FileSystem' to resource group 'Volatile'
12:50:57: OGRE EXCEPTION(6:FileNotFoundException): Cannot locate resource tagged_wall.png in resource group Volatile or any other group. in ResourceGroupManager::openResource at /home/fabien/software/ogre/OgreMain/src/OgreResourceGroupManager.cpp (line 756)

Internally, I convert a QString (UTF16) to an std::string (no encoding) like this:
QString path = ... // valid path containing non ASCII characters
std::string utf8data(path.toUtf8().constData()); // should the string coded in UTF8

These issues occurs only on Windows (compiled with MinGW gcc tool chain), but not on my Mac and Linux environments.


--- Reason of the issue ---

The standard C++ functions to open files (for example std::ifstream::open(const char *filename, ...)) doesn't support unicode (on windows at least).

Of course, wide characters can be used, but
std::ifstream::open(const wchar_t *filename, ...)
is only defined when using the VS compiler, but not the MinGW gcc compiler
Moreover it's globally not a good idea to use wide characters in C++ (ref. http://utf8everywhere.org )


--- My point of view ---

UTF8 should be supported by Ogre for obvious reasons.
Ogre should not use standard C++ functions to deal with pathes. As there is already a dependency on Boost, I'd say that using boost::filesystem to manage pathes would be much better.
The switch between std::string and std::wstring seems not useful to me.


--- Related posts ---

http://www.ogre3d.org/forums/viewtopic.php?f=3&t=48424
http://www.ogre3d.org/forums/viewtopic.php?f=2&t=47586


Could you tell me your point of view on this topic?

Sincerly
PsyCowboy

[TheSHEEEP]

According to this discussion, there should be functions like:

_wfopen()
_wreaddir()
_wstat()

that should take wchar in MinGW. Is this not the case?
Not sure if those are used in the Ogre code, though.
Your best bet for now could be to either enforce ASCII characters in your pipeline in Windows or use a different compiler there. Or, of course, change the Ogre code yourself in whatever way you fancy. If there is really something that can be done against this from my side, I doubt that I will have the time to do that in the very near future, sorry.

About the boost dependency, I think that our actual goal is to get rid of boost completely, as we only need it for threading and with the new C++11 we could use the standard library for that. Getting rid of dependencies is always a good thing if you can do an equally good replacement without the dependency. So I doubt that boost::filesystem will find its way into Ogre, as much as I like it (and I use it in almost all of my projects!)

Edit:
I forgot to add, there must be ways to deal with unicode in MinGW besides using wchar_t. There are applications out there that work with non-ASCII and MinGW.

[bstone]

I doubt that's Ogre's fault. Windows API is built around UTF-16, not UTF-8. That is the file functions like CreateFile() will not find any file using an UTF-8 path with non-ascii characters in it. On Windows you should use UTF-16, that's where the std::string -> std::wstring switch comes in.

[zarlox]

If your problem is on windows, then you can convert UTF-8 strings to UTF-16 yourself via the API:

Code: Select all

int length = MultiByteToWideChar(CP_UTF8, 0, utf8, utf8_length, 0, 0);
wchar_t *output_buffer = new wchar_t [length];
MultiByteToWideChar(CP_UTF8, 0, utf8, utf8_length, output_buffer, length);