nick.org - down!
After almost four years of no unscheduled downtime, nick.org came to a
screeching halt Sunday. I noticed an email from my brother, Kevin, indicating
that I had farked something up. "That's weird", I thought, "it was working fine
last night." I went to the website and found it very unavailable, with nothing
but a strange Apache directory listing. I attempted to ssh and found that it
was unavailable too. Uh oh. I happened to be talking to my parents via Skype at
the time (it was Mother's day) and juggling Jerry on my knee.
"I've been hacked" - was my first instinct. I started to look at other sites
that I'm hosting and found other strange Apache error messages. Oh no. I
continued to talk to my parents, figuring that I couldn't do much about it now
- but I let them know what was going on. My dad is very committed to his online
journal and I didn't want to disappoint his adoring fans for long.
I continued to poke around and found some log entries (sent via remote syslog)
that indicated a hardware failure of some sort. A double device failure? Oh no
- data loss! I grew more uncomfortable and decided I needed to tend to this
now. Eriko wanted to go downtown for some shopping and it took us a few
minutes to get ourselves and Jerry ready.
At the data center, I was confronted with a RAID controller which had gone out
to lunch. I power-cycled the machine and it believed all the disks were still
part of the RAID5 group - but once it started quotascan, the controller locked
again. My experiences at Isilon have given me quite a bit of knowledge into how
drives and controllers fail, so I decided to see if I could identify which
drive (if it were a drive) that was causing the problem. I rebooted again and
sat and watched the drive lights. Blink, blink, blink, stop. Drive 5. Stuck.
I pulled out drive 5 and rebooted. Success! I immediately rsynced a backup copy
of the data. The sites I'm hosting are mainly read-only, besides uploading
pictures, blog entries, or the log files from the websites themselves but I
hadn't yet setup an automated backup mechanism on this new server.
I left the data center with the server in a WOR (window of risk) and headed
home, glad that I had at least restored service for the time being.
On Monday, I returned to the data center with another drive. To my dismay, the
RAID controller believed it was 250.9 GB instead of 251 GB. Argh!
Reallocations! My only option was to switch to the RAID1 set (which is the boot
disk) so that the data isn't at risk. I'll rsync to the RAID5 set as well as a
remote location, which will provide some buffer until I can obtain another
disk.
Here's a shameless plug for Isilon - but boy would I have loved to have an
Isilon cluster. I could have a lost a head w/o making the data unavailable, and
I would be able to reprotect the data (at the expense of free space, which I
have plenty of) without having to immediately replace a drive. This suits me,
the busy (part-time) storage administrator, to a T. The experience left me even
more convinced that our product is light-years ahead of anything else.
