“I don’t believe in Disaster Recovery.”
What a pretty bold statement in an initial interview. The interviewer/hiring manager was a former executive of a major “hot site” company, who played a pivotal role for many clients during the World Trade Center bombing of 1993. He had a fiery reputation, and visibly stiffened at the comment.
My second sentence put him at ease, reinforced our value set, made us friends for life, and most importantly got me the job!
“Preparing for the ‘big one’ solely can leave you ill prepared for the myriad of daily events nipping at us every day. Prepare for the daily events, and use the same process for initiating processes for the ‘big’ events.”
While firms spend millions on high availability, redundancies, balancing, geographic diversity, N+1, etc., inevitably “perfect storms” lead to user impacting outages. How IT staffs respond in a crisis can differentiate companies and impact the bottom line.
The key is having everyone understand their goals, roles and responsibilities in a crisis. And during a crisis, there may be a change in an individual’s role as needs warrant.
For example, once an issue escalates to a crisis it’s useful to have at minimum two communications outlets. There needs to be management discussions focusing on impacts, alternatives, communication (to users, clients, regulators) while the technicians focus on eliminating impacts and understanding root cause.
Who is involved in each grouping is best determined in the calmness of an outage free period.
Management groupings will often include the CIO, Human Resources, Security, Public Relations, Business Continuity and the impacted business area. Technical groupings will include representatives of all the major functions, with one area tracking all the activity and logging for subsequent analysis. Frequently, the techies establish their own information communications using instant messaging or Twitter (used in a publish-subscribe manner.)
When a crisis is apparent, the notification and escalation processes have to be well honed. We often see organizations operating on predetermined voice “bridges”, with notification via text, email, phone call (in many organizations this is via an automated notification system).
There needs to be a protocol for entering the bridge, doing roll call, and general etiquette around bridge participation.
These processes are best developed in advance of need and rehearsed with staff in table top exercises.
When Information Technology leads the way in developing low overhead crisis response, and orchestrates rehearsals reducing the processes to practice, preparedness for managing the “big one” is well underway.