New OgreZip

What it says on the tin: a place to discuss proposed new features.
Post Reply
Transporter
Minaton
Posts: 933
Joined: Mon Mar 05, 2012 11:37 am
Location: Germany
x 1

New OgreZip

Post by Transporter » Fri Jun 01, 2012 2:05 pm

It would be nice to replace zzip by boost iostreams library. Boost has filter (boost/iostreams/filter/zlib.hpp) for compression based on zlib and bzip2.
0 x

User avatar
Jabberwocky
OGRE Moderator
OGRE Moderator
Posts: 2819
Joined: Mon Mar 05, 2007 11:17 pm
Location: Canada
Contact:

Re: New OgreZip

Post by Jabberwocky » Fri Jun 01, 2012 8:59 pm

One problem is that boost is an optional configuration for Ogre, whereas the zip archive is a core component. So moving OgreZip to be boost-dependent would make boost required, unless OgreZip was also moved to an optional component.

Do you have any issue with zzip, or were you just suggesting it to get rid of that additional dependency?
0 x
Image

Transporter
Minaton
Posts: 933
Joined: Mon Mar 05, 2012 11:37 am
Location: Germany
x 1

Re: New OgreZip

Post by Transporter » Fri Jun 01, 2012 9:47 pm

Jabberwocky wrote:Do you have any issue with zzip, or were you just suggesting it to get rid of that additional dependency?
I have no problems with zzip but boost is indeed a dependency on Ogre for threads. If I use boost anyway I can do without zzip. Boost is almost a standard template library as a dependency and should not be a problem.
0 x

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: New OgreZip

Post by Klaim » Sun Jun 03, 2012 1:19 pm

Yes but it's so hard to get only a part of boost included in a project that it's hard to ask any C++ dev to install it just to use your library.

That said, boost is getting modular and should be accessible officially as a tree of git repositories on github soon (like, this year).

But I don't see how Ogre can now assume that boost is a dependency. That said it wouldn't be a problem for me as I use boost in all my C++ projects anyway.
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: New OgreZip

Post by CABAListic » Sun Jun 03, 2012 2:05 pm

We've had that discussion before, and in the end it was decided not to turn Boost into a required dependency. Currently it is probably the best choice for threading support, so for most it's probably mandatory. But with std::thread standardised and supported by the more recent compiler versions it may well be that we can soon switch to that as the default threading provider, so Boost would no longer be required.

Therefore we will not currently use any more parts of Boost than the threading module.
0 x

Transporter
Minaton
Posts: 933
Joined: Mon Mar 05, 2012 11:37 am
Location: Germany
x 1

Re: New OgreZip

Post by Transporter » Sun Jun 03, 2012 3:38 pm

Maybe std::thread will become a standard but to support this you have to use new compiler. If you continue with VS2010 you have to use boost.

I use a few parts of boost threads, math and network (ASIO).
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: New OgreZip

Post by CABAListic » Sun Jun 03, 2012 3:44 pm

It is supported by recent GCC versions on all platforms and afaik on VS2011, so that should be enough to make it the default threading provider. Of course we do need a fallback for older versions, but my point is that on newer compilers we would then be completely free of Boost again, so making other parts of Ogre depend on Boost is not going to happen right now.
0 x

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: New OgreZip

Post by Klaim » Sun Jun 03, 2012 3:46 pm

That being said, VS2011 will not support XP binaries and will not be available to Xp users so it cut the users on Windows.

Also, about boost::thread, I remarked in the other thread about switching to std::thread that std::thread lacks some boost features that are interesting for Ogre. Interruptions for example...
0 x

User avatar
jacmoe
OGRE Retired Moderator
OGRE Retired Moderator
Posts: 20570
Joined: Thu Jan 22, 2004 10:13 am
Location: Denmark
Contact:

Re: New OgreZip

Post by jacmoe » Sun Jun 03, 2012 5:19 pm

CABAListic said 'to make it' which means that it will happen, just not right now.
Is that good enough? ;)

On a different note: since Windows 7 is the best Windows ever to emerge from Redmond, there is absolutely no reason for using XP these days..
((And Windows 8 is candidate for becoming the worst Windows ever - ironically. :P Let's see.. ))
0 x
/* Less noise. More signal. */
Ogitor Scenebuilder - powered by Ogre, presented by Qt, fueled by Passion.
OgreAddons - the Ogre code suppository.

Transporter
Minaton
Posts: 933
Joined: Mon Mar 05, 2012 11:37 am
Location: Germany
x 1

Re: New OgreZip

Post by Transporter » Sun Jun 03, 2012 7:36 pm

This is an interesting discussion :o
jacmoe wrote:Is that good enough?
Yes :D
jacmoe wrote:On a different note: since Windows 7 is the best Windows ever to emerge from Redmond, there is absolutely no reason for using XP these days..
((And Windows 8 is candidate for becoming the worst Windows ever - ironically. :P Let's see.. ))
I agree :D But I also agree not to drop XP, because not all companies have switched from XP to Win7. Also the upgrade from VS2010 to VS2012 need a year.

Let's get back to the topic! The main topic was the 'zzip' dependencies and not threads. One reason for my suggestion using boost is the flexible filter interface of boost. I can use zip or bzip2 with a better compression with boost. Maybe this is a topic for GSoC 2013 to implement more formats and direct use zlib instead of zzip.
0 x

User avatar
Cygon
Halfling
Posts: 70
Joined: Thu Nov 07, 2002 9:36 pm
Location: Germany
Contact:

Re: New OgreZip

Post by Cygon » Sat Jun 23, 2012 1:36 pm

I have written a clean C++ file system abstraction layer that is thread-safe and supports seamless and fast access to files stored in .zip archives without all the C cruft of zziplib.

It's got a few extras as well: there is an advanced cache system that will decide whether to directly extract data into user-provided memory or buffer it so smaller reads can be satisfied without the overhead of firing up the ZLib extraction engine each time. Oh, and I've already ported it to Linux, Windows and WinRT/Metro :)

If there is any interest, I would be willing to either donate the code or port relevant parts of it to Ogre.

You can find an overview of the design and the full source code in this blog post:
http://blog.nuclex-games.com/2012/06/th ... -archives/
0 x

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: New OgreZip

Post by Klaim » Mon Jun 25, 2012 5:02 am

That's very interesting.

How does it compare with PhysicsFS? http://icculus.org/physfs/ (other than being thread-safe)
Or more important: could you have built your library over PhysicsFS or do you see some major differences with your library implementation?
0 x

User avatar
Cygon
Halfling
Posts: 70
Joined: Thu Nov 07, 2002 9:36 pm
Location: Germany
Contact:

Re: New OgreZip

Post by Cygon » Mon Jun 25, 2012 10:49 am

I took a look at PhysicsFS before I started work on this and decided to go ahead with my own library because I disagreed with some of its design choices:
  • PhysicsFS makes assumptions about how you organize your data files. Namely, that you have exactly one big directory tree with assets that may be packaged in multiple .zip archives but which get merged again at the end.

    I believe that it is better to first take care of the platform differences completely and then built such things on top of it. I estimate that emulating PhysFS search path behavior on top of my library would take about 50 lines of code. The other way around would not be possible since PhysFS dictates the one-big-asset-directory model.
  • PhysicsFS does not handle writing well. You have to set one root path under which writes will be allowed.

    On Windows, there are several places where storing data is appropriate: C:\Users\<Name>\Saved Games for saves, C:\Users\<Name>\AppData\Roaming\<Game> for user settings, C:\Users\<Name>\AppData\Local\<Game> for computer settings, C:\Users\<Name>\Local Settings\Temp\ for temporary files. And these paths can change in location and drive letter.
  • PhysicsFS is not extensible.

    My code lets you write your own archive handlers and register them at runtime. Thus, a game can use a proprietary archive format without having to modify the sources of the library (which is rather inconvenient of the library is updated).
  • PhysicsFS is not thread-safe

    As you already noted. Thread-safety was very important to me since resource loading is a prime candidate for threading.
  • PhysicsFS has a C API.

    This is of course mostly a matter of taste, but I prefer to work with classes and exceptions in modern C++.
Other than that, both have built-in caching, both use UTF-8 in their API (both transform to UCS-2 for Windows), and both handle big/little endian issues.
+ I (biased as I am ;)) believe my library has the cleaner design and is better documented.
- On the other hand, PhysFS has been tested on many more systems than my library and can be considered mature.
0 x
Professional C++ and .NET developer trying to break into indie game development.
Follow my progress: http://blog.nuclex-games.com/ or Twitter - Topics: Ogre3D, Blender, game architecture tips & code snippets.

User avatar
Klaim
Old One
Posts: 2565
Joined: Sun Sep 11, 2005 1:04 am
Location: Paris, France
Contact:

Re: New OgreZip

Post by Klaim » Mon Jun 25, 2012 1:13 pm

Nice! I don't know if I'll be able to try it in the future but it seems very interesting.

That said, I don't see the point in integrating it in Ogre, seems like a separate library on which to build resource management systems. Or did you mean just adding an interface to easily plug it in Ogre?
0 x

Transporter
Minaton
Posts: 933
Joined: Mon Mar 05, 2012 11:37 am
Location: Germany
x 1

Re: New OgreZip

Post by Transporter » Mon Jun 25, 2012 1:23 pm

PhysicsFS supports a lot of interesting file formats like 7z. A useful strategy could be to write an abstract handler for compressed files an build the sub items only if the required libs (zlib, libbzip2, lib7z, etc.) are detected by cmake. cmake can detect different libs for threading support. Why not cloning this strategy to compressed files?
0 x

User avatar
Cygon
Halfling
Posts: 70
Joined: Thu Nov 07, 2002 9:36 pm
Location: Germany
Contact:

Re: New OgreZip

Post by Cygon » Mon Jun 25, 2012 3:04 pm

Klaim wrote:Nice! I don't know if I'll be able to try it in the future but it seems very interesting.

That said, I don't see the point in integrating it in Ogre, seems like a separate library on which to build resource management systems. Or did you mean just adding an interface to easily plug it in Ogre?
I was picking up on transporter's suggestion to directly implement zip support in Ogre.

Since I've got a complete implementation in modern C++ done already and I'm not crazy about licenses and copyright, I am offering my code, either as a bargain bin to pick pieces from or, should there be any interest, as an Ogrefication of the library itself :)
Transporter wrote:PhysicsFS supports a lot of interesting file formats like 7z. A useful strategy could be to write an abstract handler for compressed files an build the sub items only if the required libs (zlib, libbzip2, lib7z, etc.) are detected by cmake. cmake can detect different libs for threading support. Why not cloning this strategy to compressed files?
I already use an abstract handler to seamlessly integrate all kinds of archive formats into the library: ContainerFileCodec.h. Feel free to browse the classes, in terms of design and flexibility I think I'm miles ahead of PhysFS :mrgreen:

Currently, .zip support is built-in and I think, considering how small and portable ZLib is, at least that should be a fixed requirement so you have a common format that you can target safely.

As far as I can see PhysFS supports 7-Zip and Zip archives. They're struggling with bzip2 (since it's a compressed .tar). Granted, they support WAD (DooM), HOG (Descent), PAK (Quake) and GRP (Duke3D) which I probably will never add support for.
0 x

Transporter
Minaton
Posts: 933
Joined: Mon Mar 05, 2012 11:37 am
Location: Germany
x 1

Re: New OgreZip

Post by Transporter » Mon Jun 25, 2012 7:13 pm

@Cygon
Great job! Maybe I can test it this week.
0 x

Post Reply