[WIKI] Please update the static wiki once a week

[Lord LoriK]

I have my internet connection down for some time, and I thought I could grab a static copy of the wiki and the manual to carry on with my work, but I found the tarred wiki is very dated... No updates since may 2007 or so.

I know most users wouldn't care it it was updated every week, but I hope you do it anyway for the ones who do.

[Lord LoriK]

So, what's the official position about this? I've had no feedback at all, and I'd like to know what do the prominent members of the Ogre Team think about it.

[sinbad]

It's been turned off because MediaWiki has been the cause of several site-wide outages in recent times, due to some performance issues it has in certain edge cases, the static build was yet another overhead. It's low priority at least until I know we're over these issues. To be honest I've never particularly liked the idea anyway - it's out of date the second we build it anyway and is there anyone who doesn't have broadband these days? I caved in and reluctantly created the static copy but given the resources it sucks down and the performance issues with the wiki I'm strongly considering canning the static copy and telling people to live with the online version.

[SomeFusion]

I don't have broadband, and no I can't get it where I live. But the wiki works fine for me the way it is. The bigger problem for me is when users post huge .png files and videos in the showcase forums.

But ok I think I'll just have to live with that

[SpaceDude]

Why not use a web crawler like HTTrack (http://www.httrack.com/) to create your own off-line version of the site? Although this may not go down well if you suck up too much bandwidth .

[Kojack]

Or not use a web crawler if you don't want to get ip blocked (as happens on the ogre site).

(I actually like mass downloaders, I use one I won't mention to download web comics so I can read them using an image browser instead of waiting for them to download and scrolling around on the page to find the next button. With careful configuration the bandwidth hit is smaller than actually going to the site and reading them one by one, since I can filter out stuff or generate urls procedurally, or do an update so only new things are grabbed. But some sites don't like it, and will ban you. That's what happens here)

[SpaceDude]

Oh right, I thought the bandwidth hit of using a web crawler wouldn't be that bad if it was configured correctly. I can't see it being much worse than downloading a zipped version of the entire wiki. Well anyway, ignore my suggestion then unless you want to get banned it seems .

[sinbad]

The main problem with a webcrawler is that they will crawl through everything - on a Wiki, that's a major problem, as it tends to crawl through all the history and worse, all the 'diff' options. There's a lot of those, and they're not cheap on large pages. Bandwidth isn't the main issue, it's crawlers hitting diff after diff and putting the site under heavy load....

I've tried various tactics to stop this but none of them are foolproof, especially given the number of crawler tools out there. They are my arch-enemy.

Tuning continues, the situation seems to be improving but we're still monitoring to try to isolate remaining spikes. With the amount of traffic going on it's hard to see the wood for the trees and I don't have a lot of time to spend on this.

[DaMan]

It will be great if Sinbad makes a offline copy of wiki once a month or two...

[xavier]

It will be great if Sinbad makes a offline copy of wiki once a month or two...

Apart from digging up a year-old thread...

Did you not read anything he posted above? I mean, really -- a minimum of effort put forth on your part, is required to have your posts taken seriously.

[Yacoby]

The main problem with a webcrawler is that they will crawl through everything - on a Wiki, that's a major problem, as it tends to crawl through all the history and worse, all the 'diff' options. There's a lot of those, and they're not cheap on large pages. Bandwidth isn't the main issue, it's crawlers hitting diff after diff and putting the site under heavy load....

I am fairly sure their are crawlers designed for downloading wikis.
I am not sure how much it would be worth it. However, I did use it when I didn't have my internet connection for a week.

[sinbad]

Honestly, this is WAY low on my priority list. None of the tools I've tried seem to work properly, or put too much strain on the web server. Newer versions of MediaWiki require a server upgrade which goes beyond what I'm willing to undertake right now. You may just have to live with this for the rare occasions you don't have internet access, it's just too much hassle for the tiny amount of utility it brings.

Locked, that's my final word on this topic for now.