How Much Straw Can This Camel Haul?

I've been afraid to have my blood pressure checked lately. A few weeks
ago, the day I had another kidney stone attack and I learned my
grandmother had passed away it was something like 158/120. In fact, in
the last couple of months the job stress has been steadily mounting.
We closed a remote office and laid off a lot of good people and close
friends, then a few weeks later we changed our management structure and
laid off another friend - I was part of the decision making process on
both, so my stress has been high all along.

Then this past Friday was a doozie. I had a pretty good day planned,
end of the week, looking forward to the weekend. As I came through the
door just before eight I was greeted with "the web site's down".

OK, no big deal, that happens from time to time. I checked my email
and saw the site reports and a message from the ISP - the main server's
drive had crashed and they were replacing. Still no problem. I had
two separate servers, one with the FTP and SQL (the real data of the
system) and a second with the web site and processing structures -
that's the one that had crashed. I was also confident they could fix
that as well, since not two months in the past I had specifically
requested daily backups.

Or so I thought. They informed me mid morning that that drive was not
backed up. Hadn't been since July, when they changed their backup
procedures. And their backup rotation had wiped out any backups.

So I began reloading both primary sites from my machine, but that was
only HTML. I started looking for other backups and calling for outside
help. My contract programmer was on the road, but would be back in an
hour. He then called me and told me his girl friend had been in an
auto accident and he couldn't help. I called for the original site
developer, a former employee, Jamie, and left messages everywhere. And
the clock continued ticking - our online store generates on average
between $1,000 and $2,000 per day and it wasn't looking good to get
back online soon, maybe days.

Jamie left a message that he could help around 4 p.m. and that left me
fairly confident, but I soon learned he didn't have any personal
backups. I met him at the office, called in the boss and we started
tearing the place apart, looking for backup CDs. No luck. Jamie went
home to search some more and the boss kept looking. Before I went
home for medication I made a series of calls to get the drive packaged
and sent to a drive recovery specialist, a move that could cost up to
five grand.

Around ten Jamie called and let me know that basic operations were
restored, then the boss called that he had found a backup - only two
years old. With that we were back by Saturday morning. I was able to
call off the drive recovery, but we still are missing some
functionality we've added - I don't know the prognosis of getting it
back.

So, we pull another one out of the fire. Not so much fun, though.