Resource Related Functions Improvement Request

What it says on the tin: a place to discuss proposed new features.
Post Reply
User avatar
stealth977
Gnoll
Posts: 638
Joined: Mon Dec 15, 2008 6:14 pm
Location: Istanbul, Turkey
x 42

Resource Related Functions Improvement Request

Post by stealth977 »

Hi,

I have asked about this in different forums and got some answers too, but here I wanted to represent the problem from a different perspective:

The Problem: OGRE is not UNICODE

The Answer: Making it UNICODE increases the overhead too much so that the engine can run slower than NON-UNICODE version (though it is said to be possible by making Ogre::String to use wstring by some switches in OgrePrerequisites.h but it doesnt compile at least in MS VS2008)

Assumptions:
1 - Node/Entity/Material etc. generally the OBJECT NAMES dont need to support UNICODE since they are developer dependent, developer can make sure they dont need locale specific character.

2 - Only requirement is that ALL FILE OPERATIONS ARE USER DEPENDENT. We as developers do not know or can not control WHICH LOCALE users operating system uses. So lets say we created an application to display all models in a directory which is chosen by the user, if the path to this directory contains locale specific characters, then functions like addResourceLocation() fail.(They dont fail actually, but while loading any object afterwards, the resources can not be found by OGRE)

Proposal:
Only the functions dealing with FILE OPERATIONS can be changed to use STD::WSTRING instead of STD::STRING. Since currently every operating system has moved or is moving to UNICODE, this will become a must to use UNICODE capable file functions.

EDIT: Actually the main problem is with FileSystemArchieve class, it can not locate / open files when the path contains locale specific characters even if those characters are mapped to STD::STRING using locale convert functions

Best Regards,
Ismail TARIM
Ismail TARIM
Ogitor - Ogre Scene Editor
WWW:http://www.ogitor.org
Repository: https://bitbucket.org/ogitor
User avatar
stealth977
Gnoll
Posts: 638
Joined: Mon Dec 15, 2008 6:14 pm
Location: Istanbul, Turkey
x 42

Re: Resource Related Functions Improvement Request

Post by stealth977 »

Well I guess no one is interested in this issue :(
Ismail TARIM
Ogitor - Ogre Scene Editor
WWW:http://www.ogitor.org
Repository: https://bitbucket.org/ogitor
User avatar
cdleonard
Goblin
Posts: 266
Joined: Thu May 31, 2007 9:45 am

Re: Resource Related Functions Improvement Request

Post by cdleonard »

Ogre already uses too many strings for global registries of materials and scene nodes and so on. This is an acknowledged problem. Switching globally from std::string to std::wstring would make this particular problem a lot worse.

It would indeed be perfectly valid to have a separate typedef for strings used in the interaction with the file system. But this doesn't actually affect most people; it's perfectly reasonable to restrict artwork filenames. In fact until recently you couldn't even have spaces inside texture filenames (at least not inside materials). You can now use quotes (since 1.6).

This does raise an interesting problem: How do you reference unicode texture names inside ogre scripts? Switching scripting to use wstring would be very painful performance-wise. Ogre's script parsing is not particularly fast and does a lot of string operations; you might very well get close to double memory use and time spend in scripting.
User avatar
stealth977
Gnoll
Posts: 638
Joined: Mon Dec 15, 2008 6:14 pm
Location: Istanbul, Turkey
x 42

Re: Resource Related Functions Improvement Request

Post by stealth977 »

cdleonard wrote:Ogre already uses too many strings for global registries of materials and scene nodes and so on. This is an acknowledged problem. Switching globally from std::string to std::wstring would make this particular problem a lot worse.

It would indeed be perfectly valid to have a separate typedef for strings used in the interaction with the file system. But this doesn't actually affect most people; it's perfectly reasonable to restrict artwork filenames. In fact until recently you couldn't even have spaces inside texture filenames (at least not inside materials). You can now use quotes (since 1.6).

This does raise an interesting problem: How do you reference unicode texture names inside ogre scripts? Switching scripting to use wstring would be very painful performance-wise. Ogre's script parsing is not particularly fast and does a lot of string operations; you might very well get close to double memory use and time spend in scripting.
I think I couldnt express the problem very well. What I actually mean is:
* I know that using wstring everywhere is a big performance hit
* The texture names, node names, mesh names can easily be restricted to be NON-UNOCODE
* But, We have no control over PATH!!

It is very much possible the user is running on a non-english operating system and it is very much possible that the user installed a particular OGRE application on a path that contains UNICODE characters (locale specific). In this case, addResourceLocation() type commands dont work when given a fullpath instead of relative path, and thats the problem. So what I suggest is resource paths, resource adding / searching functions should use wstring, and thats all to fix the problem (the resource names can still be std::string but path must be std::wstring).

EDIT: An Example:
The user has a locale specific user name as: İSMAİL, now the users desktop path is "C:\Documents and Settings\İSMAİL\Desktop\"
The user installed the application to his desktop, opened a Game made with OGRE, played and wanted to save, so he saved his game to a folder on his desktop.
Now while loading it back, lets say that OGRE game has to add a resource location to the save folder, BOOOM, OGRE can not find any resource in this save folder since the path to folder contains locale specific characters "İSMAİL". I hope I could explain the problem better....

ismail,
Ismail TARIM
Ogitor - Ogre Scene Editor
WWW:http://www.ogitor.org
Repository: https://bitbucket.org/ogitor
reptor
Ogre Magi
Posts: 1120
Joined: Wed Nov 15, 2006 7:41 pm
Location: Finland
x 5

Re: Resource Related Functions Improvement Request

Post by reptor »

Would encapsulating Ogre3D's file operations with boost::filesystem help with this problem? I mean paths could be boost::filesystem::wpath objects.

Edit: Changed boost::filesystem::path to boost::filesystem::wpath.
soulpower
Kobold
Posts: 32
Joined: Wed May 07, 2008 8:23 am
Location: hangzhou,China
Contact:

Re: Resource Related Functions Improvement Request

Post by soulpower »

As a Chinese user , I think stealth977's idea makes absolute means. Otherwise I have to compile the source code of Ogre just to fix the Chinese patch problem
User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
x 218
Contact:

Re: Resource Related Functions Improvement Request

Post by Jabberwocky »

I just tripped across this problem too.

Up until now, I wrote my ogre.cfg file to the windows folder specified by CSIDL_COMMON_APPDATA.
On XP, this is: C:\Documents and Settings\All Users\Application Data\
On Vista/Win7: C:\ProgramData\

It turns out that C:\ProgramData doesn't work, as you need admin privileges to write into this folder in Windows7 (and maybe Vista too).
So you can't actually write out your ogre.cfg file unless you're running as admin.

There are many suggestions online about where to put writable application data, but unfortunately these are all in places that include the "user name", e.g. CSIDL_LOCAL_APPDATA, which is:
XP: C:\Documents and Settings\UserName\Local Settings\Application Data
Vista/Win7: C:\Users\UserName\AppData\Local

But here you run into the problem from the OP, which is if the username has some non-ascii character in it, then Ogre can't handle it.
I'm thinking I need to modify my local copy of ogre to use wstring filenames for the config file and logger. But in case anyone else has an easier solution, let me know!
Image
User avatar
EricB
Bronze Sponsor
Bronze Sponsor
Posts: 360
Joined: Fri Apr 09, 2010 5:28 am
Location: Florida
x 213
Contact:

Re: Resource Related Functions Improvement Request

Post by EricB »

Sorry for the necroposting...

I was wondering if either of you or anyone else has come up with an easy solution for this? Some of my users with jibberish usernames are having this same issue. :(

I did come across this thread: http://linode.ogre3d.org/forums/viewtop ... =2&t=76191 but I have no clue how he actually did it.

It's kinda funny how internationally used Ogre is and this is still a problem in 2013...
Image
Post Reply