The Insanity Must Stop

“The insanity must stop,” the newly minted IT director shouted in shear frustration.
After numerous outages caused by people making mistakes combined with some equipment malfunctions, the IT team was just beaten down. .

Too many long days, combined with long nighttime problem resolution conference calls, prompted a vocalization capturing what everyone was thinking. Saying it out loud informally made it OK to discuss the situation and even chuckle about it.

“Being lucky” often means covering the contingencies so when things go awry the organizational and computerized systems can recover gracefully. When you find yourself in a “bad patch,” don’t invocate “Murphy’s Law” as the culprit. The culprit is…you.

You are positioned and expected to lay out the processes and procedures to enable stability. A “Production Stabilization Program” is often indicated when the organization has grown very quickly and/or implemented more change than the organization/systems can tolerate.

Understanding what is going on in a “bad patch” is critical. Analyze logs and incident reports identifying common themes and root cause. Talk to your staff about what they are feeling and seeing, and what solutions they may offer.

It is often useful to categorize the situation using a cause and effect diagram (Fishbone / Ishikawa ) or borrowing from a SWOT analysis (SWOT is as an acronym for Strengths, Weaknesses, Opportunities, and Threats, categorized by internal vs. external factors.) Such a categorization allows you to visually identify the opportunity areas for the organization. On this, the tool selection is far secondary to the capturing and categorization.

With the knowledge of the impacting factors, plans can be put in place to address. Some of the fixes may be quick (i.e.: we need to reprovision some storage to address the failures) and others may become longer term, strategic initiatives (i.e.: we need to implement a disaster recovery strategy we can use “daily” mitigating outages). In this case, having a plan and implementing becomes a very sensitive issue, and one teams can rally behind.

We often hear it’s hard to be responding on a daily basis to issues and working what’s essentially a separate project to identify impacts and develop plans to address. Managers need to be sensitive when the long hours are taking a toll…when staff becomes “snappy” or “grumpy”, it’s not the best time to add work. Often a fresh set of eyes and external perspective is invaluable.

Anyone being brought to bear on Production Stabilization Programs needs to have the professional maturity and sensitivity to be perceived as not out to “shoot the messenger” or solve all the problems of the world. They are working through a separate process, and their role is about process. If the staff misunderstands this, morale will suffer substantially!

The root cause identification and articulation (at a high, process level) and subsequent planning effort can be best put together by a team external to the issues, clearly with input from the affected team, with a review and vetting by the team for management consumption.

So when you are tempted to enlist Murphy’s Law as the cause, remember what Abe Lincoln said, “when you’re being run out of town on a rail, get in front of the crowd and make it look like you’re leading the parade.” Take the bull by the horns and lead the team to stability!

Role Clarity in a Crisis

“Let Barbara do her job,” was the text message received from the CIO.

We were in the middle of a major crisis. The network had a glitch of some kind, and while the old fashioned host connected machines were fine, the Chairman wasn’t able to retrieve his email.
The conference call had been running for hours. Barbara headed (voice and data) communications, and with a deep voice background was somewhat new to data.

Since the call had run for a lengthy period, frustrations were bleeding on to the conference call. It seemed everyone was now a data communications expert, especially the desktop support people responsible for the non-disconnected clients.

So while Barbara had been leading the call, Barbara’s manager felt compelled to “help” and began directing the call, hence the text message from the CIO lurking on the conference bridge.

This brings up a couple key points in Crisis Management.

Having clarity around leadership is key. Barbara is a very competent leader, and while new to data communications was more than capable of following a process to resolution. Barbara was trying to lead her team in a structured approach AND deal with the conference call of interested parties. A more effective approach would be to have two conference calls…a technical call and a management call. Barbara should have been leading the technical call, with someone else leading the management call.

Barbara’s manager should have coordinated with Barbara were a change needed in bridge leadership. Basically taking over the bridge on strength of personality cut Barbara off at the knees. Everyone saw this (Barbara, Barbara’s staff, and the support organizations. It was not a smooth handoff, it was grandstanding unnecessary during a crisis.

Knowing who is on the call is important as well. In this case, the CIO was silently lurking on the call. It was his organization, and he was on the hook to update management. While there was no reason to exclude him, obviously it was unknown he had joined. What if the company was publicly traded and the “lurking CIO” was a member of the media? One approach some companies use is to have each conference call established with unique calling IDs (although you need to be sure ex-staff aren’t still getting the text pages).

Another uses a gate keeper to answer a call in number, confirm identity, and then join the caller with the conference call already in process. While more overhead, it also gives a chance to update callers before they join a call (as often the first question is “what is going on”, inevitably disrupting the conference call flow.

Role clarity is key in any crisis, lest a free for all develop. Clarity around leadership, management updates, protocols are all important.

We are struck by the Christmas 2009 bombing attempt on Northwest Airlines flight 253 and whether Janet Napolitano would have benefited from these lessons as she uttered, “One thing I’d like to point out is that the system worked.” The system worked after the incident, arguably there were issues before. Ms. Napolitano’s words created a separate large preventable firestorm.

Morning Operations Meeting

“Nothing productive ever happened in a meeting,” a friend once stated. He is a thoughtful guy, and his comment was not one to be idly dismissed. As you ponder this during the next meeting you attend, consider the value of a daily touch base on operational issues.

DAILY? Surely you jest.

Whether in crisis or not, a daily session is imperative in any well run operations area. And believe it or not, the meeting can be accomplished in under 10 minutes! It’s all about predictability and preparation.

Predictability

When running meetings like this, use a conference bridge with the same ID each day. Attendees shouldn’t have to search around for the contact information. Use an acronym if you can (the Morning Operations Meeting can be referenced as MOM. A conference bridge of CALLMOM (2255666) is easy to remember.

If there’s a critical mass of people at one location, try to use a conference room at that location to run the meeting. Far flung attendees participating by conference bridge is one thing, “locals” can come attend the meeting (rather than sitting at their desks reading emails!)

Pick a time when everyone can attend, based on your business day. Financial services companies will want to have the meeting well before the US stock market opens at 9:30AM (8:00 AM is a good time). If you are a retailer with stores opening at 8:00AM, an earlier time may be more appropriate.

Start the meeting on time each day. Nothing ruins the attendance and contributes to time creep than a meeting where the start time waffles. To do this, a backup chairperson should be in place to start the meeting if the chair is delayed.

The meeting should have the same agenda each day:

• Roll call


• Area by area review of any major (customer impacting) issues over the past 24 hours, with an emphasis on any active issues


• Follow up on prior action items

Minutes should be captured, and emailed to each of the areas.

Preparation

Preparation is another key to this meeting. Since the agenda is the same each day, the “areas” for review can be pre-populated on draft email. Over the 24 hours from the last meeting, Operations and the Help Desk should “contribute” major customer impacting issues to the draft. So when the meeting actually happens, the Chair is following a script of the meeting (literally reviewing a draft of the “minutes”.)

As the meeting is held, the chair can “prompt” speakers if certain issues are glossed over or missed. In this manner, major issues are not missed.

Details are not covered in this status meeting. If the issue is still active, it is placed on “follow up,” and brought back to the meeting. The chair has discretion for cutting off a discussion.

Once the meeting is completed, a brief Summary should be added to the email (suitable for reading on a BlackBerry) and the send key pressed. A wiki can also be used for this.

With predictability and preparation, the meeting will flow smoothly. Plan the meeting will run long the first week or so as people adapt to the meeting style.

Once the minutes start being read, it’s common for people to start wanting the “edit” the minutes after the fact. Some will want immediate retractions issued. My recommendation is to offer to add a “correction” section at the bottom of the minutes and issue as a part of the daily cycle. Do not get into multiple MOM minutes.

Savvy areas will want to review the “script” in advance. Why not? It allows the overall product to be stronger provided the information is factual.

And one last fun suggestion. Play into the MOM (as Mother) theme. “It’s OK to tell MOM anything. MOM is here to help.” It allows a subtle mindset shift.

And remember, you can fool some of the people all of the time, all of the people some of the time, but you can’t fool MOM.

Wanted: Technology to Drive Process

“Technology driving process; It’s not supposed to work that way.”

Anyone with formal training in process engineering knows you start with defining and optimizing your processes, and then use technology to streamline those processes. In an ideal world, this works perfectly. Layer organizational structures, system ownership and governance models, and personalities, and people naturally gravitate towards what they know best; technology.

A number of years ago I began moving my Infrastructure & Operations division to more of a process-based organization. I, and my management team, attended a series of seminars given by the late Dr. Michael Hammer. We received our certificates in “Process Mastery” and, with our newfound evangelical powers, were ready to transform the organization.

Barbara, having a reputation as an overachiever, volunteered her group, the Desktop Support, Engineering, and Help Desk department, as the first to make the move into the world of process. Over the period of a year, Barbara documented existing processes, designed new process where others were missing, created a process map, developed Service Level Agreements with users and other IT groups, and conducted training sessions with her team. The results were better delegation of tasks to the right individuals; managing to metrics, happier employees, and, most importantly, improved service levels.

As Barbara’s manager, I pushed for more improvement. Barbara responded by identifying Help Desk requests that could not be resolved on the first call and required assistance from others in the IT organization. In reviewing the list, Barbara realized 30% of the tasks could be shifted to the Help Desk and drive down costs while dramatically improving resolution time for the user. Requests such as resetting passwords, granting access to network file shares, provisioning user logins, email accounts, and printers, distribution of remote access security tokens and instructions, and creation of new user profiles were all currently being performed by senior systems administrators. The management team thought Barbara’s idea was great, and she was empowered to make it happen.

Barbara thought this would be simple. She had achieved what she thought was buy-in from the entire infrastructure and operations organization. What she didn’t expect was resistance around what people believed gave them power. Employees in other groups were fine giving over these mundane tasks so long as they still had control over approving each transaction. They felt this authority (and the trust that went along with it) is what made them special to me and others in the IT organization. What they failed to see was the erosion in the level of respect they received when they complained about not having enough resources, but failed to seize this opportunity.

The solution came through implementing technologies enabling Help Desk personnel to grant user privileges without being server administrators. Once the systems administrators saw they had not lost any control, were able to delegate tasks to the Help Desk, and were able to focus on more high value projects, they began to think about methods to offload other processes to the Help Desk. Success was achieved.

The lesson from this story is the need to discover what drives people before reengineering their processes. In this situation, the sense of control the ability to manage the technology was the drivers for the Systems Administrators. Empowering the Systems Administrators to use technology and enable the transferring of some of their processes made them supporters and eventually advocates.

In the case of the Systems Administrators, they defined their processes by technology. Therefore, technology became their driver for re-engineering their processes.

Some of the “squishier” skills, such as process, project management, client management, and budget management can be difficult for administrators and operators to understand or appreciate, and can be in conflict with their priorities. In a production IT shop, keeping systems up and running and eliminating any user downtime is the top priority. Asking people to take time away from their priorities, particularly in these challenging times, may be counter-productive and may produce defensive behavior. Take care in understanding your audience .