Resource Package Files

What it says on the tin: a place to discuss proposed new features.
Post Reply
User avatar
Brutal
Greenskin
Posts: 142
Joined: Sat Nov 07, 2009 1:51 pm
Location: Gainesville, FL
Contact:

Resource Package Files

Post by Brutal » Sun Oct 10, 2010 8:51 pm

Valve games have always come with .gcf (game content files) which is basically all of the media resources (textures/models/sounds/etc) packaged into this format. There is a tool called GCFScape that is basically a windows explorer for this package format that makes it extremely easy to view the contents, organize the files, playback sounds, view the materials, etc.

Ogre has supported loading in media from a .zip file for a long time, but it would be very nice to have some sort of Resource Package system similar to the GCF for Valve games. I hesitated requesting this as it is more of an addon than a feature, and in this case seems rather game specific instead of rendering engine. However, all Ogre projects use media content of some sort, and having something similar to this probably wouldn't be a bad idea. Compression and Security are not necessarily required, as it is more for organization. However, valve does use a "CTX Key" which is basically a keycode hardcoded into the engine required to open there GCF files.

The creator of GCFScape has actually released the source code which is quite interesting to look over at http://nemesis.thewavelength.net/index.php?p=35
0 x
**Certified Ogre Professional Amateur

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Resource Package Files

Post by CABAListic » Sun Oct 10, 2010 9:15 pm

Since I'm not familiar with gcf, exactly what do you see as its advantages over a zip archive?
0 x

User avatar
betajaen
OGRE Moderator
OGRE Moderator
Posts: 3447
Joined: Mon Jul 18, 2005 4:15 pm
Location: Wales, UK
Contact:

Re: Resource Package Files

Post by betajaen » Sun Oct 10, 2010 9:25 pm

I always thought a one material/one texture/one mesh per zip file was a good idea, problem is the amount if disc access it needs.
0 x

reptor
Ogre Magi
Posts: 1120
Joined: Wed Nov 15, 2006 7:41 pm
Location: Finland

Re: Resource Package Files

Post by reptor » Sun Oct 10, 2010 11:50 pm

Okay betajaen that was funny :lol: you got me laughing. Thanks.



More seriously, I think if you want to have a package file format then it's not terribly difficult to create one by yourself.

You can lay it out something like this... you'll see from this that this is not really complicated at all. This is approximately how such a binary file format could be designed, starting from the beginning of the file:

1) Package file signature, example "NUMB", have this as an unsigned int in code so you won't be checking for a string when you open the file and want to know if it's likely your package file format.

2) Package file version number, unsigned int, 1 and so on, you know.

3) Now the next thing is a list of file entries you have stored into the package file. You can associate some "meta data" with the file names like a last write time stamp, file size when not compressed, a bool for if the file is compressed or not, file size when compressed, and so on... maybe a checksum of some sort of the original data if you like.

4) OK so now we have the list of files. The next thing is to write each file sequentially into the package file. Compressed or not, it's your choice.

5) The end! Nothing here. The package file ends at the end of the last file that was stored into the package file.

This is not difficult to implement. So if you don't like using ZIP or some other common package file format then creating your own format is not hard.

Some further ideas for such a file format. It could have "file groups" so that you could have for example a file group called "text files" and all of those text files would be stored together into the same package file data block with compression. This would mean that the compression ratio could be improved as compressing the data blocks separately produces a bigger total output size versus compressing them together as one block. This would mean that you would have to load the whole group at once but this could be used for files that you know are most likely needed at the same time in your application or the cost of having them all in the memory would not be too much so irrespective of whether your application needs them or not you could just load them all up front.

Going even further, thinking about a tool to create such a package file, you could have it detect when identical data blocks are coming in and either stop there and tell you about it or only store one data block in the package file but have separate entries for them in the file entry list. I have noticed that some compression tool that I use on Linux is dumb regarding this - if I have two or more identical files somewhere in the file system tree that is being compressed into one archive file, it will actually store them separately in the archive file which can be a huge waste of space. Of course I could try to make sure my file system doesn't have identical files but really I would expect a compression tool to notice it and not store several identical data blocks into the archive file - it is enough just to write the file entry information separately for each but not the data itself. For a game package file creation tool this would be helpful to notice when you are accidentally trying to store identical files in your package file. Actually for some media types (CD/DVD on consoles) they can want to store identical files many times over to make the loading from the physical media more sequential but let's keep this out from this.

OK let's try to somehow get back on topic. The benefit of such a package file is that your application would only request one file to be opened by the operating system. This obviously has performance benefits. And anti-virus software can increase the cost of opening files so this advantage really should not be underestimated. Even if you do no compression at all to the files it's still better than having them all as separate files.


Any ways, I guess my main point is that this is not hard to do from scratch by yourself. If you want to tie your file loading code into OGRE you could even use some classes from OGRE to help you with this (personally I avoid doing it as I don't want this code in my projects to depend on OGRE but this is very much a personal choice).
0 x

User avatar
betajaen
OGRE Moderator
OGRE Moderator
Posts: 3447
Joined: Mon Jul 18, 2005 4:15 pm
Location: Wales, UK
Contact:

Re: Resource Package Files

Post by betajaen » Sun Oct 10, 2010 11:59 pm

reptor wrote:Okay betajaen that was funny :lol: you got me laughing. Thanks.
What? I was serious.

I'm not suggesting it should be used for every resource file you have, it's just a nice alternative for small games, addons, DLC or mods. Big resource files still have their place, and are very useful, especially on consoles when everything is on DVD.

Having a material/mesh/texture in one compressed file is very neat, just copy it into your resources, later on it can be replaced with a newer version, and it allows modding. The only downsize is you have tons of files in your directory, and the time opening and closing many files is slower than just doing it once on a huge file.
0 x

reptor
Ogre Magi
Posts: 1120
Joined: Wed Nov 15, 2006 7:41 pm
Location: Finland

Re: Resource Package Files

Post by reptor » Mon Oct 11, 2010 12:21 am

I honestly thought that was a joke. But I believe you if you say it wasn't :P

"The only downsize" you mentioned is actually a big downsize. It is big-enough to justify using a package file for all game content.

And I can't see how a package file of all game data would disallow modding. If you want people to mod your game then give them a tool which unpacks and packs the files. And you can let them define what should be loaded, perhaps in addition to the original data, or perhaps over-riding the original data. Of course this can be configured any way - it does not matter how the files are stored.

One of the biggest gripes I have about some of my favourite games is the loading time! This annoyance shall not be underestimated.

Oh, and as an example, some applications I use parse tons of XML files every time on start-up... that's nuts! I am never going to modify the files so why can they not optimise them for loading performance. It's like the user of the application suffers because the developers liked the XML file format during development of the application. This is not how it should be - by all means use a slow but flexible method during development, but don't force it down my throat when you ship the final product to me. This surely concerns some games too.

Modding is nice but it doesn't have to be done this way - just give them a tool to unpack and create package files and that solves it. Oh, and please no XML parsing... at least not of big files or of many files. That's a very verbose file format. Please for the love of dog optimise the file formats.

Let's look at it from another angle. Every user will do a "pre-processing" step every time they start the program that should have been done at development time. This will waste massive amounts of energy with popular products.
Last edited by reptor on Mon Oct 11, 2010 12:28 am, edited 1 time in total.
0 x

User avatar
betajaen
OGRE Moderator
OGRE Moderator
Posts: 3447
Joined: Mon Jul 18, 2005 4:15 pm
Location: Wales, UK
Contact:

Re: Resource Package Files

Post by betajaen » Mon Oct 11, 2010 12:27 am

GTA uses massive files for content, and it's always been a pain to add and change files. A good way would be to literally copy the small resource zips into an override folder, with some sort of XML file explaining what the mod is.

Like I said, large resource container files are very useful, but not very mod friendly. Unless your given the tools to do, then if you distribute your mod, other people will need those tools to mod their game with your files. It's a pain really, and copying a few files into a folder, always seemed simpler to me. Fallout 3 and Oblivion do it, and they have a HUGE modding scene, and both use .bsa files; massive resource archives.

Anyway, I've said what I wanted to say.
0 x

reptor
Ogre Magi
Posts: 1120
Joined: Wed Nov 15, 2006 7:41 pm
Location: Finland

Re: Resource Package Files

Post by reptor » Mon Oct 11, 2010 12:33 am

But you can have several package files if you want to mod a game... I mean if you create a game and want to allow people to mod it. Of course in such a case you don't expect them to mod the original data files but to create their own mod package files. Then what they need to do to let others play their mods is to give them the mod package file and the game would have a mechanism to perhaps automatically detect such extra files or let the user point the files to it. This is not a problem, really. People wanting to play the mods would not need any tools. If you create a game you can design it the way you like, right? So if you want to allow modding and playing with the mods, you will make it easy to do. I honestly don't see a problem.

I can see your point of view that the big bulky file can be more difficult to handle, but really modding is the exception and not the rule - most people do not mod the games so why would you compromise the data file loading speed for the sake of a few people who want to mod the game to let them have easier access to the original game data files.
0 x

User avatar
Brutal
Greenskin
Posts: 142
Joined: Sat Nov 07, 2009 1:51 pm
Location: Gainesville, FL
Contact:

Re: Resource Package Files

Post by Brutal » Mon Oct 11, 2010 7:46 am

I just skimmed over the responses as I just woke up, but in response to the "annoyance of adding files to large content packages", valve has a nice system for this as well. Basically the games come with the file structure of the resource files outlined in the game directory. For example...

[BaseGameDir]
--->[ModelsDir]
--->[MaterialsDir]
--->[SoundsDir]

These directories are basically empty though. However, if a file is added to them that is the same as one of the files in the GCF, it loads the one from the folder instead. This basically provides an easy way for people who want to use custom models, sounds, etc. to simply drag drop them in. So if models/player/bob.mesh was in the GCF file, and a user put a modified version of models/player/bob.mesh into their game directory, it would use that instead.
0 x
**Certified Ogre Professional Amateur

User avatar
betajaen
OGRE Moderator
OGRE Moderator
Posts: 3447
Joined: Mon Jul 18, 2005 4:15 pm
Location: Wales, UK
Contact:

Re: Resource Package Files

Post by betajaen » Mon Oct 11, 2010 8:30 am

Oblivion and Fallout3 also do this, which is why I love them. What I wanted was a quicker way that you just throw a zip file in there, and the program would be smart enough to know to use those resources, instead of using a tool to replace a file in a huge resource container file.
0 x

CABAListic
OGRE Retired Team Member
OGRE Retired Team Member
Posts: 2903
Joined: Thu Jan 18, 2007 2:48 pm
Contact:

Re: Resource Package Files

Post by CABAListic » Mon Oct 11, 2010 8:41 am

PhysFS is the answer, my friends ;)
0 x

reptor
Ogre Magi
Posts: 1120
Joined: Wed Nov 15, 2006 7:41 pm
Location: Finland

Re: Resource Package Files

Post by reptor » Mon Oct 11, 2010 6:26 pm

Well, it's one choice :) I choose to use my own solution though and it's pretty simple so I think my solution is more light-weight as I don't need that many features. But PhysFS is good too, I have tried it.
0 x

User avatar
SunSailor
Gnoll
Posts: 699
Joined: Sun Jan 02, 2005 5:45 pm
Location: Velbert, Germany
Contact:

Re: Resource Package Files

Post by SunSailor » Mon Nov 15, 2010 1:08 am

If you are interested in a commercial release, I can tell you, that my online distribution system called Jade:DS has such resource system itself, but obviously, it is for the distribution over my platform only.
You can see a demonstration of the system and the rest of the integration in this thread. There is a full integration into ogre available and used in the example.
0 x
Online-Distribution with Jade-DS, Ogre-Wrapper included.
Follow me on Twitter for updates!

User avatar
Zonder
Ogre Magi
Posts: 1133
Joined: Mon Aug 04, 2008 7:51 pm
Location: Manchester - England
x 22

Re: Resource Package Files

Post by Zonder » Tue Nov 16, 2010 2:58 pm

I don't think specific formats should be included with ogre but there is no harm in plugins for the feature. I would agree with CABAListic about PhysFS looks very nice I know they say it's video game targeted but who wouldn't want an app that can be modded easily after release I'd vote for support there instead :)
0 x
There are 10 types of people in the world: Those who understand binary, and those who don't...

Post Reply