Hybrid Harddisk performance

[syedhs]

I know some of you may be scratching your head while waiting for your compiler to finish linking (omg, since when compiler link? but as long as it is understood ) because linking stage can be easily the most harddisk intensive activity. And solid state harddisk is probably the answer, but it still way too pricey, at least for me.

But I just bought myself a hybrid harddisk which Seagate claims as hybrid between solid state and conventional harddisk. You can read about it here.. : http://www.seagate.com/www/en-us/produc ... laptop-hdd. I am not sure but other manufacturers could have something similar in their offerings.

Anyway, I bought one - 500GB version and I definitely I can feel the performance increase, link process is probably sped up by 3x. Yes 3 x, but I should say I didn't run any test using timer but I remember I could go to the kitchen, made myself a cup of coffee and it was still not halfway done - so this is a subjective comparison. Compilation stage performance increase is probably not noticeable though as you barely can see harddisk light constantly lit up (as opposed to druing link stage). So if you have some cash to spare and you are doing C++ a lot, then upgrading slow harddisk (5400PM?) to this hybrid harddisk is definitely worth it.

[Kojack]

That's interesting. It's a 500GB 7200rpm hard drive with 32MB cache and 4GB of nand. The nand memory holds frequently (or recently) accessed files from the drive, as a kind of large persistent cache.

However there's a potential problem that would have me a little unsure about them. One of the biggest flaws of ssd drives is that the nand memory has a limited number of writes before it wears out. Reading is fine, but writing kills the thing over time.
Using nand in a hybrid drive as a cache would mean that it's contents are being changed depending on what you are reading. So even if you just do a ton of reads, the nand is still doing writes as it caches in different files. So I'd expect it to not last as long as an ssd drive (unless you always read the same 4GB worth of stuff).
Although that's assuming that the memory it's using has the same properties as a typical ssd drive. Maybe they can afford to stick better quality stuff in there since there's only 4gb of it.

Still sounds interesting. Anything to speed up compile times is cool with me.

[spacegaier]

I just completely re-setup my PC yesterday featuring a new hardware addition:

A 120GB Corsair SSD !

Suddenly, everything feels lightning fast, not only compiling/linking but also simple tasks as opening documents and such. A real performance boost! So I can definelty understand syedhs, althought my former main drive (and now slave drive for data ) was already not a bad one (7.200RPM and 32MB cache).

Regarding the lifetime of the SSD: I read that now with Win 7 and enhanced drivers, the SSD can make sure that each part is used equally and therefore enhance the lifetime. So I am not worried too much about that point.

PS: Bought it for 200€, so still much more expensive than a regular HDD, but already in theupper, affordable range for the mass market IMHO.

[Kojack]

Just be careful, compiling big things like ogre is pretty much the worst thing (besides playing minecraft) that you can do on an ssd, since it involves huge numbers of obj file writes. It would be great for something like all the system headers, boost, etc, but bad for the intermediate and output directories.
I was thinking of getting maybe a 300GB velociraptor, but for the price I could get 2-4TB of normal sata drives.

[spacegaier]

True, but it fells so fast .

Will have to monitor the usage/lifetime via some S.M.A.R.T. tools. Seems that Corsair doesn't provide one of its own.

[betajaen]

I really don't understand why modern OSes use RAM disks more. Memory is cheap, and could be easily allocated for caching common files, and would be considerably fast, more so than a SSD. I mean, is it really that difficult to get a few sticks shove them onto some sort of PCI board, and let the operating system use it like any other drive? Hard-drives are slow and clunky, and SSDs are fast but have limited write life. You could use SSDs to store those files whilst the machine is off, and transfer operating system into memory on boot up.

On the Amiga, the RAM disk came as standard and was used by the OS to cache preferences and temporary files. This was fairly unique, since an Amiga typically had only 2 megs of RAM to work with. It used to resize when more space was required, and you could even boot from it. This from an Operating System made in the 80s. Hardly taxing for more modern computers, isn't it?

[_tommo_]

I really don't understand why modern OSes use RAM disks more. Memory is cheap, and could be easily allocated for caching common files, and would be considerably fast, more so than a SSD. I mean, is it really that difficult to get a few sticks shove them onto some sort of PCI board, and let the operating system use it like any other drive? Hard-drives are slow and clunky, and SSDs are fast but have limited write life. You could use SSDs to store those files whilst the machine is off, and transfer operating system into memory on boot up.

I am wondering the same... today is cheap to get a machine with 8gb of super-fast DDR3 memory.
So why i can't just tell the filesystem to use 4 of them as a super-fast cache for files?
I think ReadyBoost on Windows Vista/7 does exactly that, but for unknow reasons only if you spare a dedicated NAND device for that.
I know that there are also programs that let you use RAMDisks, but the ones I tried looked somewhat buggy, and the whole system got hiccups and freezes while accessing files.

[reptor]

I proposed the idea of using a RAM disk to speed-up a really demanding 3D game. People shot the idea down by ridiculing my understanding of how operating systems work and other such arguments which looked more like very emotional responses than facts.

Then a year or two later some guys actually tried the idea and found it improved performance substantially. I was like "no sh**, that's exactly what I said would likely happen..." It's to me an obvious solution to slow loading of content from hard disk drives! And people told me it can't be because of how operating systems work. Well, the operating systems are not that smart as was proven.

[Kojack]

Ogre compiles really fly when you stick it in a 4GB ram drive.

Although even without that there's things that can be done to speed compiles up, like when I changed Ogre to use a Unity Build (no relation to the competing 3d engine). A full release compile of ogremain dropped from over 9min to just 41 seconds.

I haven't tried unity builds into a ram drive though, that could rock.

[_tommo_]

... I changed Ogre to use a Unity Build ... A full release compile of ogremain dropped from over 9min to just 41 seconds.

Please, do explain

[betajaen]

Have a look at this; http://buffered.io/2008/06/19/screencas ... ty-builds/

I've been using it in NxOgre for a month and a bit now, basically it can compile NxOgre in 6 seconds now, where it used to take a few minutes.

[_tommo_]

Already found this, and I was going to edit my post

Big thanks for the hint guys, it is really much faster also when compiling small projects... I wonder why this isn't the default choice of any modern IDE.
to when an Ogre build that uses this by default?

Just a question: what about incremental builds?

[spacegaier]

to when an Ogre build that uses this by default?

That would indeed be great.

[betajaen]

https://github.com/betajaen/buildbot/tree/master/unity

Spacegaier just twisted my arm to post that. Basically it's the "Unity Build" code from BuildBot, which I use in NxOgre. It's a ruby script that creates a ".cpp" file to use with Unity Builds.

You configure it via the last few lines of unity.rb; "cpp_files" is the list of files that are your source code, "target" is the file to write to, and "source" is the what the file should look like. The configuration is YAML based, so watch your indentation.

[spacegaier]

Without ever actually having tried UB (will do in the next hours/days), this could be a great addition. We once would have to run that script and add the output to the repository and everytime a new CPP file is added to the repo the script needs to run again.

If that is really everything that needs to be done (and it is if I am not missing something) it sounds like next to no work for a huge improvement in build time for everyone without even the need to care about the topic since we already provide the buildbot result (perhaps we could guard it via a CMake option).

[CABAListic]

Actually, I'm convinced that this can be implemented in CMake alone, no external tools required. (Which is very preferrable.)
There are, however, potential downsides to Unity builds, so this would required extensive testing, particularly if the gain really is that impressive consistently. On my quadcore, compile times are already at <3mins. After all, Ogre already has a lot of build options, I'd rather be cautious when adding another one.

[Kojack]

One thing I did that's different to the standard unity build (I really wish it didn't have unity in the name) is I didn't put all the cpps into a single unity cpp. Instead I split the files over 4 cpps. There are two reasons for this:
- vc++ can do parallel cpp compiles. With a single cpp I had just over a minute to compile, using 4 cpps with 4 threads dropped the time down to 41sec.
- Ogre has a couple of global symbols defined multiple times (which is naughty), arranging the files so the multiple definitions aren't in the same unity cpp meant I didn't have to edit ogre source to get it working (although I think there was one line still needed, I can't remember).

I also had to add the /bigobj flag to the compile options, ogre is pretty damn huge.

[ekt]

Ogre compiles really fly when you stick it in a 4GB ram drive.

kojack i'm interested in this
do you mean something like http://techreport.com/articles.x/9312

or a software ram disk? i'm sure in the past i've read that - quite shockingly - a ram disk would perform slower with vs compilation times

[Kojack]

It was a software ram drive.
I had a trial one, I copied all of ogre to it and compiled, it went damn fast (it was something like 5-10x faster). But of course you then have the overhead of copying the source to the ram drive, then copying the end result back to a permanent drive. But for frequent recompiles (like testing ogre config flags, they hit nearly every header so cause complete rebuilds) it's cool. Just don't leave your only copy of the source in ram.

I've got a new one installed on here: Dataram Ramdisk. It's free for up to 4GB and works in windows 7 64bit.
It's not enabled right now, but it worked ok. I haven't tested compile speed.

[Mikachu]

Actually, I'm convinced that this can be implemented in CMake alone, no external tools required. (Which is very preferrable.)

I just read that blog post which uses CMake for this purpose...

[merlinblack]

Hmmm Ram drive eh? I might try some Linux tomfoolery and see what happens. Linux basically uses any "unused" RAM as file buffers anyhow, so I wonder if I could get away with just the build tree in the RAM drive.....

[_tommo_]

Actually, i found out that RAM disks have quite no effect on compilation times - barely visible on a huge build like Ogre - while if builds were IO bound they should be even faster that Unity Builds (n accesses to disk vs no accesses).

Probabily modern compilers/IDEs are smart enough to keep a .h file in RAM until the whole build is finished; in fact in VS++ you can edit and save files while building with no problems.
So, maybe this "I/O-boundness" of the builds is mostly and urban legend, and each file is loaded once in any case.

then, why unity builds and precompiled headers have this huge effect over compilation time?
I think that it's because the compiler has to parse and compile the headers each time it finds them referenced, so more accesses = more compiler work, and C++ compilers aren't exactly fast.
So building anything once and only once greatly helps the compiler in skipping redundant work.

Short version: RAM Disks won't help (that much)

[CABAListic]

On a related note, has anyone ever tried using ccache on Linux, and if so, how does it fare when (repeatedly) compiling projects like Ogre?

[Zonder]

back to the original post

Have you tried HDTune on that to see what MB/s it gets? SSD in raid can get as much as 500Mb/s (See the Sony VAIO Z11). On a side note how long do you guys expect a HDD to last normal ones the reacon under office usage is 3 years before they fail (personally I only trust a year!) I am sure SSD can get the same with windows 7

[reptor]

I have a hard disk drive from 2001 and it works fine.

I have never had a hard disk drive fail and I've had a PC since the early 90's. And I use my computers too much...

The only problem I have had was when I damaged a power cable connector. It was entirely my fault and the HDD returned to normal behaviour after I had fixed the connector.