Munu Emergency Blog

Saturday, August 27, 2005

Fsck!

The server has decided that it has to do a full filesystem check right now. This normally does this once every six months after a reboot. And today is the lucky day.

This does mean that the server isn't broken, but there are something like a million files on there (thanks to the busy Munuvians), so it will take a little while to run. Probably on the order of an hour, and we're half an hour into that already.

Oh yeah: A filesystem check on Unix (or Linux) is done by a program called fsck.

Update: Coming up now. Sorry about the delay.

Bugger

That didn't work. The system will be back shortly, but we will have to schedule yet another outage to finish the work. :(

Dum De Dum

The second disk drive is being replaced right now. We'll be back any moment... Any moment now...

Friday, August 26, 2005

The Blog Ate My Homework - I Swear...

Funny thing happened on the way to publishing a post this morning. I hit publish, it published, but it didn't post. I dutifully refreshed the page & nothing. I rebuilt the entire site - nada. The entries are still on my editing page & saved but for some reason they just won't post to the site. I ended up posting the two entries at my back up site (publicola.blogspot.com)but I'm still puzzled.

Has this happened to any of ya'll? Ladies tell me do you understand?
And all my fellas do you feel my pain?

Planned Outrage: Saturday 27th August

The server will be down for about an hour beginning 5AM CST while the other hell demon disk drive is exorcised replaced.

Hopefully that will fix all of our present problems, so we can get on with creating new and better problems.

Thursday, August 25, 2005

Damn And Double Damn

We are now getting error messages from the other disk drive. If you've ended up here, it probably means the server is down while we replace that disk as well. It shouldn't take so long this time, since everything is already backed up.

I'm going to try rebooting and see if that helps at all. If we can make it to the weekend I can schedule the second hard disk to be replaced so that there's a minimum of downtime, but if it gets really bad I'll have them replace it right away.

Wednesday, August 24, 2005

And As The Blog Sinks Slowly In The West

This blog will now drift peacefully off to sleep, hopefully for a very long time.

Back In Business!

We're up and running! Yay!

It might be a bit slow for the first few hours because the RAID array is rebuilding, but it is definitely working and nothing has been lost.

Thanks for your patience everyone! Now my bed is calling me. :)

We'll Be Back

MuNu isn't dead, just pining for the fjords. When I'm done putting 10,000 volts through it, it will voom good as new.

There doesn't seem to be any way to sticky posts, so I kicked this into the middle of next week.

(If you were looking for something other than MuNu, such as a forum or a portal site, then it was hosted on the MuNu server and will be back along with everything else as soon as we can possibly manage it.)

Good News!

The cheesy disk drive has been kicked to the kerb! I repeat, the cheesy disk drive has been kicked to the kerb!

The techies are testing things now, but it's looking good.

Okay. Nearly There...

Everything is across onto the backup server, and all the main databases have been restored. A few databases for individual users wouldn't load (including one important one) but I can get most things running now. The databases are from yesterday, but I managed to copy all the individual files across, so we hardly lost anything.

But!

Tech support at the hosting company is replacing the failed disk right now. (Yay!) If that works, then we can just go back to our happy MuNu-ing. If it fails, I will bring MuNu up on the backup server and off we go.

Hang tight for another hour or so, kids, we're nearly there!

Titles? We Don't Need No ....

Eh. Blogger. Can't really bitch too much about it, but I'll try anyway. :P Actually it's not that bad. It'd be nice to have an extended entry so I don't take up too much space bitching/not bitching about blogspot, but beggars can't be choosers.

I'll be honest: my knowledge of things PC & internet is mainly pre-jurassic. So when like most of you I tried to click on my site I was a bit perplexed. Hell, I figured the ATFU finally shut me down. Then I checked some other Munuvians & discovered that it as system wide. I still don't know exactly what went wrong, but then again I can't claim I knew exactly what goes right 99.995% of the time.

Confederate Yankee offers to post the URl of any Munuvians on his back up site. E-mail him at confederateyankee AT gmail DOT com. (Replace AT with @ & DOT with . respectively).

Basil is accepting trackbacks of Munuvian back up sites at http://www.basilsblog.net/2005/08/mu_nu_where_are.html

The lovely & talented Beth is offering to let Munuvians post on her site & she as well is accepting trackbacks. Click here for the relevant info.

Many thanks to all three of those fine young bloggers for lending a hand. Or more approriately Yay! (what the hell does that mean anyway??) Aftet the revolution their names will be on the list of Friends of the People.

& thanks to Pixy who is busting his ass trying to get things up & running again so we can not post for days at a time but still pull up our pages hourly in a vain attempt to get "inspired".

Finally a message to all our Munuvian brothers & sisters trapped without a convenient way to post:

The chair is against the wall. The chair is against the wall.
Babygirl think she cute. Babygirl think she cute.
Cadilacs can only swim through epsom salted water. Cadilacs can only swim through epsom salted water.

That is all.

Update: 08-24-05 07:09 MST

That is not all. I have received word from our...well let's just say I have recieved word. Words rather; there was more than one involved. Now the message:

The jaded eagle flies at midnight. The jaded eagle flies at midnight.

You know what to do. Carry on.

Damn And Blast

The database restore failed somewhere near line 2.8 million of the 9.8 million line backup file.

Which is bad.

I've just cooked up a little Python program to split the backup file into individual databases so I can restore them one at a time. We'll see if that works a bit better.

Update: Okay, off and running again. That's the good side of dumping to an ASCII format - you can hack it up if you have to. (The bad side is that you sometimes have to.)

Sleep, That Wits The Knavelled Reave Of Care

I'm going to see if I can grab a couple of hours of this elusive commodity while the database restore runs. Everything seems to be going well... Relatively speaking.

...

Ah crap, the restore fell over. :(

Meh

Gave up on trying to dump the SQL databases again; it was going to take something like 20 hours. I've kicked off the restore from yesterday's backup, and that is going to take long enough by itself.

Fun fun fun...

Trying To Back Up MySQL

It's sort of working. Not fast, though.

Timing Is Everything

Our hosting company's tech support website seems to have gone offline. Nice.

Homeward Bound

I'm heading off home now, so I'll be offline for an hour or so. Just in case something goes wrong...

Nevermind.

Tuesday, August 23, 2005

You Know...

Blogger doesn't suck as much as it used to.

Compared to Movable Type on a server with a stuffed hard disk, for example.

Rsyncing Feeling

Well, it's doing better than expected in that it hasn't died completely. Every so often it just freezes for a minute or three. But I do want to get all the stuff that was posted since the last backup, so no choice really.

Suckiness

It takes forever just to update that little place-keeper post on munu, so further updates will be found here.

If that rotten disk would just drop dead, everything would be fine again. Sigh.

MuNu Done Blowed Up Real Good

Hi munu fans.

The munu server has blowed up. The various problems we've been having with spam floods and denial-of-service attacks and getting Farked have been real, but the reason they were killing the server was an intermittent drive failure.

That failure is no longer intermittent.

Our last full offsite backup was yesterday, so this could have happened at a worse time. I'm working on recovering any updates from the last few hours, and after that I'll be bringing things up on our backup server. This may take a few hours, so please bear with us.

(Yes, we have mirrored disks. If one of them had simply failed outright, this wouldn't have been a problem. Instead, it's gone completely mad, generating spurious errors and generally screwing things up.)

Update 16:13 AEST: All user files and critical system files are on the backup server, and I'm attempting to bring it up to date with rsync. I've logged a request for the faulty drive to be replaced; at this point we don't have much to lose even if they get it wrong and pull out the good drive...

Minor update: The rsync is doing much better than I'd expected, so I'll let it run. After that, I'll try for a backup of MySQL so we'll be completely up to date.