Have a contingency plan and know when to activate it.
I worked in an Information Technology department for many years, and this was one of our mantras. So it's been interesting to deal with contingency planning myself over the past two days.
Here's what those past two days have looked like for me, with approximate times:
Saturday, 6 p.m.
I came home from working with a client, went to my computer, and checked e-mail - and when no messages came in, I knew something was wrong. I went to my web site - and there was no web site! Horrors! But I could browse the web just fine.
So I called the company that hosted my web site, and was told there was a significant data center problem, but everything would be fixed by midnight. I wasn't happy - that would mean almost 11 hours of downtime, since the problem had started a bit after 1 p.m. - but I went to bed confident that all would be fine in the morning.
Sunday morning, 8 a.m.
Still no e-mail, no web site. Now I was told things should be fixed by 7 p.m. The phone support person did all he could do - he acknowledged the problems this extended downtime caused for me, and gave me the best information he had - but that didn't make the downtime situation any better.
Sunday morning, 8:30-noon:
No longer trusting the estimates, and unable to accept this extended downtime, I went searching for a new hosting company that met my requirements, which include around-the-clock phone support in the U.S. Fortunately, I already had some leads that I'd noted from prior reading.
Sunday noon:
I placed my order with my new hosting company.
Sunday, early afternoon:
I did all the technical work required to get my site moved and e-mail running again. Fortunately, I understood what needed to happen, and that went pretty smoothly. There was one glitch, but having a friend test my new site uncovered that one.
Sunday, 3 p.m.
My site came back up, at least for some users. E-mail started flowing again.
Sunday, 5:00 p.m.
My new hosting company sent me a welcome message on Twitter.
Sunday, 5:15 p.m.
My original hosting company provided updated information; service was now expected to be restored by 7 a.m. on Monday.
So what did I learn from my web site hassles? Being organized for such an eventuality really helps! Here are the basics of being prepared:
1. Have a plan.
I could have saved about four hours if I knew exactly what hosting company I would move to if mine ever had a catastrophic failure. But having some leads was much better than starting from scratch.
2. Review and modify the plan.
While I'll pick a "next in line" hosting company now, I know I need to review that decision periodically; companies change, for better and worse.
3. Know when to act on the plan.
While I hadn't put my decision criteria into words, I think I made some good choices here. I made the leap when:
- I no longer trusted the information coming from my old company.
- The probable downtime from staying put was long enough that the probable downtime associated with moving seemed like the lesser of the evils.
And these same lessons apply to other areas of our lives - which can be much more important then a web site problem. For example, I've had similar issues when one of my cats got sick on a holiday weekend. Where do I go when my vet is closed? When do I decide it's time to go, rather than wait for my normal vet to be open?
We know we need plans to deal with major issues - a fire, a serious earthquake or hurricane, a medical emergency. (We may not have those plans, but we're aware of the need!) But it's easy to overlook the more mundane blips in our lives, and make sure we're reasonably ready for them, too.
Sounds like we have the same hosting company - it's nearly 10 am EST Monday and everything is still down. I'll be looking too, but for one that has a better backup plan of their own.
ReplyDeleteWe might indeed, Larry!
ReplyDeleteYou raise a good point about the hosting company itself having a backup plan. Having a huge part of your data center go down would be a tough one to be prepared for - although they really should have some sort of plan for that. Here's what's happening at my old company:
"During an annual fire system inspection of our data center the fire suppression system was inadvertently triggered and discharged into the data center environment. The discharge of the fire system has affected the hardware and operations of our servers."
OMG!!! That's cuh-razy! Would you mind saying who you went with and, if you dare, who you left? I need to work on my backup plan!
ReplyDeleteThanks, Jeri, your blog is the oldest blog on my netvibes page. Tried and true.
Nancy N.
Nancy, I had been using WestHost, and I really loved them. They had 24/7 technical support in the U.S., and the phone support people were superb. I'd had a few short periods of downtime, but nothing that bothered me too much.
ReplyDeleteBut this incident has been a horror. People are as mad about the lack of communication from management as anything else. While many of the servers are back now, some may be out for as much as 96 more hours - and it seems this is just now being acknowledged.
Info here:
http://netstatus.westhost.com/
http://members.westhost.com/outages.php
Community discussion here:
http://forums.westhost.com/forumdisplay.php?f=144
I switched to HostGator. The company has the 24/7 US-based phone support I want, and seems to get good reviews. There was no set-up fee, and it's a month-to-month agreement - so if I change my mind, I can easily move again with no financial penalty. But so far, so good.
The good old computer. Friend and foe at the same time. Have had a few mishaps myself and hopefully have learnt a thing or two out of these painful lessons. I like to backup my important files periodically. It's also good idea to store your electronic files offsite if you can.
ReplyDeleteAnd if you work at home, treat your disaster preparedness strategy with the same seriousness as a company would.
What an adventure! It's the technical part that would slow down most of us. I'm glad you were able to handle it well. Under those circumstances, my options would be to hyperventilate and eat chocolate!
ReplyDeleteCynthia, I'm lucky that I have a bit of a technical background, so I can handle that part OK, with a bit of assistance from the hosting company's support staff.
ReplyDeleteAnother thing I did right: The company where I registered my domain is NOT the same company that hosted my web site. If they had been the same, I wouldn't have been able to make the move.
DebraC, backups are indeed crucial. In this whole incident, I was lucky in that I have all my web site files on my laptop (which, in turn, is backed up); I just needed to upload the files to a new server.
ReplyDeleteI've written about backups before; here's one such post.
Always reminds me of the "Jesus Saves" analogy.
ReplyDeleteStrange I should stumble across this (via http://administratosphere.wordpress.com/2010/02/23/downtime-reports-how-did-they-respond/)
as we speak the server my website is hosted on has been down for approximately 12 hours. It is just a personal site and e-mail that's impacted and frankly I'm getting what I paid for, but doesn't make it any the less frustrating. As a professional sysadmin I just increasingly find myself tempted to pay for a cloud VPS and do it all myself.
Thank you so much for taking the time to answer my question, Jeri!
ReplyDeleteNancy N.
Nancy, thank YOU for commenting - and for the kind words.
ReplyDeleteGarp, I'd forgotten all about that old Jesus saves thing!
I definitely don't want to manage my own server. But I'd be willing to pay extra for a company that had great tech support on the phone, and a really good disaster recovery plan. I'm not looking for the cheapest hosting company - but I don't need many of the services that come with higher-priced options. There don't seem to be many companies targeting someone like me.