Insight #46 : Finding the Edge: Risk Management in an Era of CautionDisaster Recovery and the Cloud in the Wake of Hurricane Sandyby Ned Schoenfeld
One of the most expensive natural disasters ever to hit the United States, Hurricane Sandy cost the country an estimated $63 billion. More than 265,000 businesses in New York State were affected - and 72,000 homes and businesses in New Jersey were either damaged or completely destroyed. In the aftermath of the storm, millions of employees were shut out of work for days at a time, their offices closed because of power outages, a lack of heating, and communications failures. For Ned Schoenfeld of Pcubed, many of these ongoing business interruptions could be avoided if disaster recovery strategists embraced the potential of cloud applications instead of focusing on vulnerable physical locations.
"Before Hurricane Sandy many organizations believed the main goal of disaster recovery planning was to have a back up copy of all data held off-site," Ned Schoenfeld explains. "After Sandy however, it has become clear that you actually need to have the capability to run the entire business off-site."
For Schoenfeld, Managing Director of Pcubed's New York Office, the failure of many organizations to cope with the kind of conditions created by Hurricane Sandy can teach important lessons in how to face the disasters of the future.
"In many cases," he continues, "large corporations based in New York were simply unable to return to their buildings for days at a time. The offices were either flooded, or lacking basic necessities such as heating, electricity, or the internet. To keep operations going at all, organizations were scrambling to run alternative decentralized work places using cell phones.
"Hurricane Sandy really has caused a major shift in thinking about how companies should prepare for situations when they can't physically access their offices."
The aftermath of the cataclysmic storm which hit the eastern seaboard on October 29, 2012 caught many firms off guard; it was not the kind of disaster scenario they had planned for or anticipated. For Schoenfeld, being denied access to a physical workplace should not be a reason for an organization to put its operations on hold.
"Businesses have seriously underspent on thinking through this kind of scenario and overspent on the physical data centre part. One large New York bank, for example, had built an office building in the city for 10,000 of its employees - but had then chosen a disaster recovery site which was directly across the street. When the whole street flooded, and no one could get into either of the buildings, they were in real trouble. The business was unnecessarily compromised because no one had anticipated having no access to their physical office location."
The solution is to make it possible for employees to be able to work effectively from any location - independent of a vulnerable central server.
"Certainly from a technological perspective cloud applications provide a good solution to this problem," Schoenfeld says.
"If you're using Gmail as your primary email system, when a hurricane hits, your email server won't go down and you won't need a back-up facility.
"With Microsoft Office 365, for example, the host infrastructure is spread all around the states, and indeed the world. It gave businesses a huge advantage when the infrastructure on the East Coast was knocked out. The area affected by Sandy was so huge and caused such a widespread network outage, it took out many organizations' back up facilities as well as their primary servers. If much of a company's software, email, and communications are cloud-based - then much of that risk is eliminated."
Introducing a virtualized business desktop frees staff from their ties to the physical office environment. "Employees can use their cell phones to communicate and just need to borrow any computer in the U.S. with Internet access to be able to do their work effectively," Schoenfeld says.
Introducing a cloud-based system does not negate the need for other logistical disaster recovery plans, however. "It's important to work out a plan for managing day to day business activities if the phone system is out, for example," he explains. "You need to work out the details of how to contact all your staff, how to enable them to conduct business meetings, and to work through the essential documents you need to get online. All of these issues should be fully worked through long before the plans are needed."
So what are the key steps businesses wanting to mitigate the loss of a physical office location with a successful disaster recovery plan need to take?
"First, businesses need to rank their processes in order of priority, and by the length of time an outage in the process can be tolerated," Schoenfeld says. "For example, corporate books and ledgers are a high priority process, but one that could tolerate a three-day outage. This level of hiatus could not be tolerated in a sales or customer services situation, on the other hand, so both of those processes would be defined as a high priority and in need of the fastest possible recovery time.
"The next step is to figure out the most efficient and lowest cost method of altering the current process to make it geographically independent, e.g. moving software which had previously been hosted internally onto the cloud.
"It is possible to run a separate data center off-site" he continues, "but if the telecoms in the primary office are down, then you have no access to it. This was a really common problem during Sandy - many corporations were relying on separate data centers to carry them through, but without Internet access in the primary office, they could not connect to them.
"The key question is to identify the least expensive way to eliminate this dependence. By using the cloud, the work is redistributed so your ability to process is not based in one single geographical area that can all be affected by a widespread telecoms outage."
Once those fundamentals are identified, planners need to look at how and when individuals across the business will be given access to the cloud server, and how to implement the change in business continuity change. "This needs to be assessed in terms of the cost impact to the business," Schoenfeld says.
"The organization also needs to decide on the level of quality it is willing to accept in its back-up operational processes. Is 85 percent of functionality acceptable for disaster recovery mode? Not all businesses would need 100 percent to tide them over until normality returns - 60-70 percent might well be enough."
From then onwards, the transition can be managed in the same way as any other large-scale project. Once in place however, it requires regular real world testing. "You can't find out on the day the hurricane hits that the system doesn't actually work," Schoenfeld points out. "You need to schedule one week every year to run all systems in disaster recovery mode to make sure it works, and to give the organization confidence. Managers need to make sure everyone knows how it works - appreciates its strengths and weaknesses, and knows exactly what they need to do when it's needed for real."
The huge cost and scale of the havoc wreaked by Hurricane Sandy in October has forced the question of disaster planning to the forefront of many people's minds. The question for businesses - as with individuals - will always be about how to find the right balance between exposure to excessive risk - and costly over-protection.
"Business continuity is like buying insurance," Schoenfeld says. "You don't want to buy too much, as it's expensive - but if you buy too little, you may find you're not covered when you actually need it.
"The good thing about cloud technology is that a relatively small amount of money is required to solve the need for a physical location, the key issue for many businesses affected by Sandy." The surge of interest in less traditional ways of approaching disaster planning should help many organizations into a much stronger position for when the next disaster strikes - in whatever form.
Pcubed has helped many clients execute cloud based technology programs in recent months including for a major international telecom provider. For more information on how Pcubed can help your organization transition to cloud based strategies, please contact Linda Lavine, Pcubed Director of Marketing, at email@example.com.
Ned Schoenfeld formerly ran Pcubed's New York office until leaving us in 2013. He also held executive positions at UBS, Citibank, and other financial services and healthcare organizations.