nick.org

Jerry turned two today. Eriko and I are both amazed at how quickly the time has passed and how much he has grown. I'll post some pictures as soon as I upload them.

• The happy family.


• The cake eater!

nick.org - down! After almost four years of no unscheduled downtime, nick.org came to a screeching halt Sunday. I noticed an email from my brother, Kevin, indicating that I had farked something up. "That's weird", I thought, "it was working fine last night." I went to the website and found it very unavailable, with nothing but a strange Apache directory listing. I attempted to ssh and found that it was unavailable too. Uh oh. I happened to be talking to my parents via Skype at the time (it was Mother's day) and juggling Jerry on my knee.

"I've been hacked" - was my first instinct. I started to look at other sites that I'm hosting and found other strange Apache error messages. Oh no. I continued to talk to my parents, figuring that I couldn't do much about it now - but I let them know what was going on. My dad is very committed to his online journal and I didn't want to disappoint his adoring fans for long.

I continued to poke around and found some log entries (sent via remote syslog) that indicated a hardware failure of some sort. A double device failure? Oh no - data loss! I grew more uncomfortable and decided I needed to tend to this now. Eriko wanted to go downtown for some shopping and it took us a few minutes to get ourselves and Jerry ready.

At the data center, I was confronted with a RAID controller which had gone out to lunch. I power-cycled the machine and it believed all the disks were still part of the RAID5 group - but once it started quotascan, the controller locked again. My experiences at Isilon have given me quite a bit of knowledge into how drives and controllers fail, so I decided to see if I could identify which drive (if it were a drive) that was causing the problem. I rebooted again and sat and watched the drive lights. Blink, blink, blink, stop. Drive 5. Stuck.

I pulled out drive 5 and rebooted. Success! I immediately rsynced a backup copy of the data. The sites I'm hosting are mainly read-only, besides uploading pictures, blog entries, or the log files from the websites themselves but I hadn't yet setup an automated backup mechanism on this new server.

I left the data center with the server in a WOR (window of risk) and headed home, glad that I had at least restored service for the time being.

On Monday, I returned to the data center with another drive. To my dismay, the RAID controller believed it was 250.9 GB instead of 251 GB. Argh! Reallocations! My only option was to switch to the RAID1 set (which is the boot disk) so that the data isn't at risk. I'll rsync to the RAID5 set as well as a remote location, which will provide some buffer until I can obtain another disk.

Here's a shameless plug for Isilon - but boy would I have loved to have an Isilon cluster. I could have a lost a head w/o making the data unavailable, and I would be able to reprotect the data (at the expense of free space, which I have plenty of) without having to immediately replace a drive. This suits me, the busy (part-time) storage administrator, to a T. The experience left me even more convinced that our product is light-years ahead of anything else.

Less than 3 years ago, I was the proud owner of a fully populated entertainment center/bookshelf. This is not one of those small pieces of furniture, mind you, but at least 10 feet in length and 8 feet tall.

I love information - I'm essentially an information pack rat, although as you'll see, that is changing. I've kept most of my books from college - especially my math and computer science books. I really (read never) have time to read them, but just seeing them on the shelves takes me back to a time where I was younger, smarter, and more in control. =)

First came Eriko. I gave up 25% of the bookshelf as soon as her stuff began to arrive from Nagoya. There was some creative maneuvering and I ended up with more shelf space, by far.

Then came Jerry. At first, the impact was minor. A shelf for his toys. Then two. Then a shelf for his books. Then two. Then another shelf for his toys. Then another for his books.

I put quite a few of my books in storage - all the "duplicate" (or redundant) materials, or the ones of least interest to me - but I still kept quite a few CS, math, and economics books - not to mention my collection of Dr. Dobb's
magazines (which were doubling as a barrier to accessing the power cords.)

As of last Sunday, I was down to slightly less than 50% of the total space - but looking around the room, at the proliferation of toys and books, I finally bent to the forces of family.

I boxed up all my magazines (Dr. Dobbs and the Economist) and literature books, moved all of my CS books to work (where they probably won't be read either, but I wasn't using those bookshelves), and consolidated my math, economics, and Japanese language books onto a single shelf.

The funny thing is that I find the event quite freeing. I can't count how many evenings I would grab one of those CS books, read a page or two, and feel tortured - I couldn't spend time with it but I couldn't leave it alone. This reminds me of the time, in our condominium garage, when one of my neighbors came up to me. I was "working" on the MG and she said "when are you going to grow up and just get rid of this thing? You should be using this space for the car you park on the street."

She's right - I knew it then and I know it now, but I still haven't given up that car (and I still entertain ideas about converting it into an electric car for Jerry). I don't have too many things left to go. =)

Zack had been prompting me to read this for some time, so I took it with on my trip to Texas last week. It turned out to be very easy and engaging reading and I finished it on the flight down. Reading Wil's blog entries and then his own commentary on the blogs was interesting. I have found a very similiar duality or "show" which occurs in my own writing.

When I first started this blog, I was writing it for me - my earliest entries are intimate, revealing, and sometimes "inappropriate." As my audience appeared and it became clear that Eriko, my parents, my siblings, and other relatives were reading the blog I found myself altering the type of content and presentation.

No longer do I write an entry as a stream of consciousness in a vacuum - I'm very aware of the "audience" watching. At the same time, my life has changed so much, most of my content has become less brash; there is no need for such bravado - gone are the days of political and international conspiracy theories (who has time for that), the partying (day after is too painful), and the women (one is very much enough, thank you).

My entries are focused on career and my family. I find it fitting that this matched Wil's transition as well - he is my role model for the "goals" page on nick.org.