Lessons from Hurricane Irene or how are your BCP/DR plans in practice?

So the media was all over Irene and with good reason.  At this writing, 21 people have perished in the hands of this evil lady, with hundreds of millions (or more) in damage.

In my lovely town, we greeted Irene with caution.  Sure, I had bought a couple tarps and some duck tape (yes, mine says duck, and not duct.)  After all, at 29 miles west of the ocean, I wasn’t concerned about storm surge.  I did invest $36 in a DC/AC inverter as an afterthought with the intention of streaming video from my moving F350 truck. 

Fortunately, it never got bad enough in this neck of the woods to “do foolishness.”  Nearly all my town of 18,000 lost power at 10:49AM Sunday….and this gave time to consider technology and DR/BCP plans.

As I sat with my neighbor on my front porch last night at 11PM, enjoying a crisp, post-storm sky and watching the police slowly patrolling for evil-doers, my neighbor kept saying, “What was it like without power?”

Well, what was it like without power?

I contend we were never really completely without power.  Everyone had cell phones, and they were clearly working for the duration.  AT&T, Verizon, MetroPCS and Sprint all seemed fine.

A last minute Livestream account afforded me the opportunity to “broadcast” live from the scene.  Frankly, the scene was (fortunately) pretty boring.  When the power went out, the laptop’s battery kicked into gear, and a battery operated mifi unit kept me on the air.

A local gas station had to close when the power went out.  The pumps wouldn’t work. 

The local pizza house, accustomed to producing hundreds of pizzas a day using an electrically operated conveyor oven, had to use the original “old school” gas fired pizza oven, creating a backup of as much as two hours for pizza (and making some customers gripe loudly.)  They used flashlights and emergency lighting.  Since the cash register was down, battery operated calculators were used.  With no phone service, the only ordered was at the restaurant, and credit card numbers were written down for subsequent processing.  Was this part of their formal disaster recovery plan?  No…it was put together on the fly.

On those customers griping loudly….it’s interesting to observe crowd behavior.   The storm had been forecast for a week, with public authorities recommending people “stock up” on a “couple days” food.  While two hours is a long time to get a pizza, when you look into the darkened kitchen seeing the staff working with flashlights and the owner manning the pizza oven…somehow “starvation” doesn’t come to mind.  Patience and understanding is what comes to mind.  One thirty something patron showed the bright side when she declared, “the beer is still cold” while waiting for the pizza.

Without power, the WAN connection to my home failed.  As a Verizon FiOS user, with fiber optic to the home, I tried powering the router with the inverter and could not pull a signal  (the pole mounted repeaters were powerless.)  My old school copper phone did work until the FiOS battery backup failed HOURS into the event.

That $36 inverter provided power for the laptop, and fluorescent lighting in two homes.  While it was nice to have some light, and to stay connected, a Disaster Recovery strategy of truly working from home would need to address any VoIP phone issues (perhaps using soft phone), and longer battery times. 

Monitoring the police/fire scanner, the fire department was kept busy dutifully responding to reports of fire alarm failures throughout town.  As the batteries lost power in these systems, they arranged for a “one last call”….creating a subsequent follow up task for the fire department.  It strikes me these systems should have at least a 6 hour battery backup if not more.

Companies investing in appropriate infrastructure or co-lo weathered the storm quite nicely, and I couldn’t find any popular site or bank not responding during the height of the storm.   On the front deck we could hear the muffled generator from a distant cell tower, enabling our communications.  My town hosts a number of large data centers, and they were impervious to the storm.

The MBTA shut down service before the storm hit.  While protecting their staff and systems during the storm, I’m always reminded how many people rely on public transportation to get to the office.

At 9:49PM, my cell phone rang with a robo call from my electric company, suggesting they were working on safety first (downed lines) and then would work to restore normal power.  “It may be several days before all power is restored.”  They even have a nifty website for seeing outages.Sample Power Outage Website

At 7:21AM, my power was restored, and in my corner of the world things are quickly getting back to normal.  Honey Dew Donuts is still without power and Dunkin’ Donuts is picking up the slack.  So I’m a lucky one.  My thoughts go quickly to those who lost loved ones and/or homes.  We have an expression, “never let a good disaster go to waste”, and Irene presented this central Massachusetts town with lessons.

Companies of all sizes and types can work to ensure they have solid business continuity and disaster recovery plans, and that the plans are regularly tested.  Making plans up “on the fly” is good for an adrenaline rush, and not good for business.  Investing in the appropriate infrastructure, and understanding how communications are provided to your facility (routing, including above or below ground considerations) are key.

In the Northeast, Hurricanes are rare.  Yet “stuff happens” and daily little outages do take place almost daily.  We’ve put together business continuity and disaster recovery plans and performed assessments for startups to large investment management companies.  If your company doesn’t have this, it’s time before the next event.

How did you and your company weather the storm?