New OgreZip

[Transporter]

It would be nice to replace zzip by boost iostreams library. Boost has filter (boost/iostreams/filter/zlib.hpp) for compression based on zlib and bzip2.

[Jabberwocky]

One problem is that boost is an optional configuration for Ogre, whereas the zip archive is a core component. So moving OgreZip to be boost-dependent would make boost required, unless OgreZip was also moved to an optional component.

Do you have any issue with zzip, or were you just suggesting it to get rid of that additional dependency?

[Transporter]

Do you have any issue with zzip, or were you just suggesting it to get rid of that additional dependency?

I have no problems with zzip but boost is indeed a dependency on Ogre for threads. If I use boost anyway I can do without zzip. Boost is almost a standard template library as a dependency and should not be a problem.

[Klaim]

Yes but it's so hard to get only a part of boost included in a project that it's hard to ask any C++ dev to install it just to use your library.

That said, boost is getting modular and should be accessible officially as a tree of git repositories on github soon (like, this year).

But I don't see how Ogre can now assume that boost is a dependency. That said it wouldn't be a problem for me as I use boost in all my C++ projects anyway.

[CABAListic]

We've had that discussion before, and in the end it was decided not to turn Boost into a required dependency. Currently it is probably the best choice for threading support, so for most it's probably mandatory. But with std::thread standardised and supported by the more recent compiler versions it may well be that we can soon switch to that as the default threading provider, so Boost would no longer be required.

Therefore we will not currently use any more parts of Boost than the threading module.

[Transporter]

Maybe std::thread will become a standard but to support this you have to use new compiler. If you continue with VS2010 you have to use boost.

I use a few parts of boost threads, math and network (ASIO).

[CABAListic]

It is supported by recent GCC versions on all platforms and afaik on VS2011, so that should be enough to make it the default threading provider. Of course we do need a fallback for older versions, but my point is that on newer compilers we would then be completely free of Boost again, so making other parts of Ogre depend on Boost is not going to happen right now.

[Klaim]

That being said, VS2011 will not support XP binaries and will not be available to Xp users so it cut the users on Windows.

Also, about boost::thread, I remarked in the other thread about switching to std::thread that std::thread lacks some boost features that are interesting for Ogre. Interruptions for example...

[jacmoe]

CABAListic said 'to make it' which means that it will happen, just not right now.
Is that good enough?

On a different note: since Windows 7 is the best Windows ever to emerge from Redmond, there is absolutely no reason for using XP these days..
((And Windows 8 is candidate for becoming the worst Windows ever - ironically. Let's see.. ))

[Transporter]

This is an interesting discussion

Is that good enough?

Yes

On a different note: since Windows 7 is the best Windows ever to emerge from Redmond, there is absolutely no reason for using XP these days..
((And Windows 8 is candidate for becoming the worst Windows ever - ironically. Let's see.. ))

I agree But I also agree not to drop XP, because not all companies have switched from XP to Win7. Also the upgrade from VS2010 to VS2012 need a year.

Let's get back to the topic! The main topic was the 'zzip' dependencies and not threads. One reason for my suggestion using boost is the flexible filter interface of boost. I can use zip or bzip2 with a better compression with boost. Maybe this is a topic for GSoC 2013 to implement more formats and direct use zlib instead of zzip.

[Cygon]

I have written a clean C++ file system abstraction layer that is thread-safe and supports seamless and fast access to files stored in .zip archives without all the C cruft of zziplib.

It's got a few extras as well: there is an advanced cache system that will decide whether to directly extract data into user-provided memory or buffer it so smaller reads can be satisfied without the overhead of firing up the ZLib extraction engine each time. Oh, and I've already ported it to Linux, Windows and WinRT/Metro

If there is any interest, I would be willing to either donate the code or port relevant parts of it to Ogre.

You can find an overview of the design and the full source code in this blog post:
http://blog.nuclex-games.com/2012/06/th ... -archives/

[Klaim]

That's very interesting.

How does it compare with PhysicsFS? http://icculus.org/physfs/ (other than being thread-safe)
Or more important: could you have built your library over PhysicsFS or do you see some major differences with your library implementation?

[Cygon]

I took a look at PhysicsFS before I started work on this and decided to go ahead with my own library because I disagreed with some of its design choices:

• PhysicsFS makes assumptions about how you organize your data files. Namely, that you have exactly one big directory tree with assets that may be packaged in multiple .zip archives but which get merged again at the end.
  
  I believe that it is better to first take care of the platform differences completely and then built such things on top of it. I estimate that emulating PhysFS search path behavior on top of my library would take about 50 lines of code. The other way around would not be possible since PhysFS dictates the one-big-asset-directory model.

• PhysicsFS does not handle writing well. You have to set one root path under which writes will be allowed.
  
  On Windows, there are several places where storing data is appropriate: C:\Users\<Name>\Saved Games for saves, C:\Users\<Name>\AppData\Roaming\<Game> for user settings, C:\Users\<Name>\AppData\Local\<Game> for computer settings, C:\Users\<Name>\Local Settings\Temp\ for temporary files. And these paths can change in location and drive letter.

• PhysicsFS is not extensible.
  
  My code lets you write your own archive handlers and register them at runtime. Thus, a game can use a proprietary archive format without having to modify the sources of the library (which is rather inconvenient of the library is updated).

• PhysicsFS is not thread-safe
  
  As you already noted. Thread-safety was very important to me since resource loading is a prime candidate for threading.

• PhysicsFS has a C API.
  
  This is of course mostly a matter of taste, but I prefer to work with classes and exceptions in modern C++.

Other than that, both have built-in caching, both use UTF-8 in their API (both transform to UCS-2 for Windows), and both handle big/little endian issues.
+ I (biased as I am ) believe my library has the cleaner design and is better documented.
- On the other hand, PhysFS has been tested on many more systems than my library and can be considered mature.

[Klaim]

Nice! I don't know if I'll be able to try it in the future but it seems very interesting.

That said, I don't see the point in integrating it in Ogre, seems like a separate library on which to build resource management systems. Or did you mean just adding an interface to easily plug it in Ogre?

[Transporter]

PhysicsFS supports a lot of interesting file formats like 7z. A useful strategy could be to write an abstract handler for compressed files an build the sub items only if the required libs (zlib, libbzip2, lib7z, etc.) are detected by cmake. cmake can detect different libs for threading support. Why not cloning this strategy to compressed files?

[Cygon]

Nice! I don't know if I'll be able to try it in the future but it seems very interesting.

That said, I don't see the point in integrating it in Ogre, seems like a separate library on which to build resource management systems. Or did you mean just adding an interface to easily plug it in Ogre?

I was picking up on transporter's suggestion to directly implement zip support in Ogre.

Since I've got a complete implementation in modern C++ done already and I'm not crazy about licenses and copyright, I am offering my code, either as a bargain bin to pick pieces from or, should there be any interest, as an Ogrefication of the library itself

PhysicsFS supports a lot of interesting file formats like 7z. A useful strategy could be to write an abstract handler for compressed files an build the sub items only if the required libs (zlib, libbzip2, lib7z, etc.) are detected by cmake. cmake can detect different libs for threading support. Why not cloning this strategy to compressed files?

I already use an abstract handler to seamlessly integrate all kinds of archive formats into the library: ContainerFileCodec.h. Feel free to browse the classes, in terms of design and flexibility I think I'm miles ahead of PhysFS

Currently, .zip support is built-in and I think, considering how small and portable ZLib is, at least that should be a fixed requirement so you have a common format that you can target safely.

As far as I can see PhysFS supports 7-Zip and Zip archives. They're struggling with bzip2 (since it's a compressed .tar). Granted, they support WAD (DooM), HOG (Descent), PAK (Quake) and GRP (Duke3D) which I probably will never add support for.

[Transporter]

@Cygon
Great job! Maybe I can test it this week.