Telehouse, The Human Element: Not an Electron Dropped
September 21, 2016
Business Continuity During Hurricane Sandy
Hurricane Sandy was already a dangerous storm when it first gathered force in the Caribbean Sea off the coast of Nicaragua. It ultimately wreaked devastation upon the Jersey Shore and the entire New York metropolitan area as it roared up the northeastern seaboard and reached landfall on October 29, 2012. Today, “Superstorm Sandy” stands as one of the deadliest and second-costliest hurricane in U.S. history, taking 72 lives, mostly in New York and New Jersey, and causing an estimated $75 billion in damages.
A significant but unheralded story about Hurricane Sandy was the performance of Telehouse’s data centers and the people who kept them at peak performance, 24/7, during the event. As Fred Cannone, Director, Strategic Sales and Channel Development at Telehouse, notes in an interview, “We didn’t drop a single electron of power or communications due to our preparation, our ability to maintain a controlled environment, and our people.”
This was true for all three of Telehouse’s New York locations, which literally and figuratively were in the eye of the storm and yet functioned without interruption.
When Milliseconds Can Mean Millions
In a hyper-connected world, where even milliseconds of downtime can cost companies millions of dollars, enterprises’ as well as SMB’s ability to operate even in the worst of times is not a business nicety but a necessity. Fifty percent of businesses that experience data loss for 10 days or more file for bankruptcy immediately, and 93 percent fail within a year. That’s why having a comprehensive Disaster Recovery plan in place, one that is continuously refined, frequently rehearsed and coordinates all stakeholders and partners — including data center operators, fuel and parts suppliers, power and communications service providers, and customers — is truly mission-critical. Indeed, Disaster Recovery preparedness, which starts with accessible data backup, is the foundation of business continuity. This was exemplified by the steps taken by Telehouse to ensure that when Sandy hit, its operations and those of its customers were not adversely impacted.
Preparation is Everything
Telehouse New York Teleport opened in 1989 and comprises 162,000 square feet of colocation space. Located just 17 miles from Manhattan in a 100-acre office park on Staten Island, and operated by the Port Authority of New York and New Jersey, its concrete and steel structure was built to withstand 125-mile-per-hour winds. Sandy’s winds struck the Island at 80 miles per hour. Interestingly enough, given its address, Teleport is not located in a flood zone, but because it sits on a dry topographic prominence above shore, is afforded natural protection from high water surges.
Teleport features two completely diverse fiber routes to and from the facility with diverse building entry paths and multiple bandwidth and IP connectivity options. It also has six diesel-powered generators in an N+1 configuration and a 48-hour diesel fuel supply on-site, with deliveries from multiple vendors available within 24 hours.
The latter point is noteworthy, since during Hurricane Sandy and two hours before there were electric outages across New York caused by preemptive shutdowns of the public utility’s electrical service in order to preserve the integrity of the system, Telehouse went off the public utility’s electrical grid and used generators to keep its data centers running continuously for five days, thereby avoiding any instability or uncontrolled spikes with the public grid.
For an anticipated disaster like Sandy, additional attention was given to ensure spare parts were on hand if necessary and that all fuel tanks were topped-off for the backup generators. Importantly, Telehouse’s Disaster Recovery planners also sourced extra fuel from vendors outside of the projected storm path to avoid fuel contamination from flood waters.
Long before the hurricane reached landfall, Telehouse worked with customers to make sure they knew what to expect, the procedures to follow and the people to contact. In addition, customers received status updates every two hours for the duration of the emergency.
Every Story Has Heroes
Teleport data center personnel were deployed for 10 to 12-hour shifts and, according to protocol, properly provisioned to work for days and, if necessary, possibly weeks on end. While the data center’s power and telecommunications systems worked without interruption as planned, as Cannone calls out, this outcome would not have been possible were it not for the facility’s people.
“Our engineers, including Dave Kinney, Director of Facility Engineering and Planning, and our customer service personnel, were the real heroes during Sandy,” he comments.
Grateful customers agreed. Keith Heckel, VP, IT Infrastructure, at Mitsui Sumitomo Marine Management said at the time, “I would like to congratulate all those that were involved in keeping the Telehouse Staten Island facility fully functional during Hurricane Sandy. Not once did we experience a disruption in connections or service during this event. Telehouse has proved that it is a first-class provider for colocation services. This was a well-planned, executed and communicated disaster recovery effort. This was all accomplished even as the Staten Island area was dealing with a direct hit from the hurricane.”
No Matter What, Just Another Day at the Office
Telehouse and its parent KDDI have pioneered educating customers on the criticality of mitigating vulnerabilities during a natural or man-made disaster by having a detailed Disaster Recovery and Business Continuity plan in place. As Cannone remarks, “Calling upon over 25 years of experience, we align IT and facilities capabilities with our customers’ business objectives so that every day, no matter the circumstances, is just another day at the office.”
Four years after the event, the repair of the damage done by Sandy’s winds, and particularly its tidal surge, are still ongoing. As some will remember, the storm’s raging waters affected New York’s subway system and the entire region’s transportation network. Trains didn’t run for days, tunnels and stations across the system were flooded, and signals and switches were fried. Repairs are planned through 2020.
Meanwhile, in its updated forecast, the National Oceanic and Atmospheric Administration predicted this hurricane season will be the most active since Sandy hit in 2012, meaning the need for reliable Disaster Recovery and Business Continuity solutions has never been more critical.
