Oct 30, 2015
How Datto Protected Business Data during Hurricane Sandy
This week marked the third anniversary of Hurricane Sandy, so we’re looking back at how Datto protected our partners and their end users, and how important business continuity and disaster recovery are at all times, especially during natural disasters.
Hurricane Sandy was truly a huge test for Datto. Prior to Sandy, the biggest test was Irene in 2011, but Datto had yet to experience anything on this scale until 2012. Not only was Datto’s corporate headquarters in Norwalk in the direct path of the hurricane, but one of the company’s data centers was also affected by the storm.
As we noted in our look back on Hurricane Katrina, it comes down to being properly prepared.
In Datto’s case with Sandy, we informed partners of the proper steps they needed to take before the storm hit to keep themselves protected and one step ahead. This included confirming their systems were ready to be virtualized off site if needed.
As the storm was approaching, one of Datto’s partners was very behind on his backups and needed to get his SIRIS device synchronized off-site immediately. Traditionally this issue would be solved with a round trip drive (a USB hard drive that is used to take the backups off of a partner’s local Datto device, which is then shipped to Datto’s headquarters to be synced directly with Datto’s cloud.) With Sandy looming, there was simply not enough time to ship the drive to the partner and back to Datto in time to sync the device. A technical support representative went the extra mile and drove to meet the partner at a midway point between Norwalk, Connecticut and the partner’s site to intercept the round trip drive. He then rushed the drive back to Datto’s headquarters where it was uploaded into the cloud, saving the partner from potentially crippling downtime. Datto then began preparing their employees, corporate office and data center to weather the storm.
A full crew of Datto employees was assembled to man Datto’s Cloud Operations Center and technical support staff throughout Sunday the 28th and the morning of Monday the 29th to handle preparation efforts for their clients. That night, Datto had a full team of support representatives, Systems Administrators, the VP of Operations, Director of Technical Support and even CEO Austin McChord on staff. The team ensured that operations were never interrupted throughout the storm and partners always had assistance available. Fully stocked with enough food and supplies for three days, Datto Support was ready for the long haul.
“(Our response was) a testament to the unwavering dedication of Datto employees to assist our Partners through this disaster.” Austin McChord, Datto CEO
In the event that Datto’s HQ lost power, a backup plan was in place. A remote Datto Disaster Recovery Team was sent to Datto’s east coast data center in Reading, Pennsylvania, where technicians established a fully operational, remote extension of Datto Technical Support. The remote team’s primary responsibilities were to handle overflow of high-level disaster recovery tickets and ensure continuity of technical support in the event of power outage or bandwidth loss at Datto’s corporate HQ. Boyd McCorkle, a member of the DSS team, remarked: “We were ready at a moment’s notice to take on full operations should HQ go down. [Our command center] made for a virtually seamless transition.” Datto was ready.
During the storm, the keys to working efficiently were teamwork, seamless communication and the dedication of Datto’s staff. Partners were able to get in contact with technical support directly where tickets were solved immediately or quickly escalated to Datto Cloud Operations, or the Disaster Recovery Team located in Pennsylvania. Due to the large number of Datto employees available, DR virtualizations were handled as fast as they came in. An open Google Hangout between the remote Disaster Recovery Team and the Cloud Operations Center facilitated continuous communication, which allowed tickets to be triaged and supported as fast as possible.
One of Datto’s partners, Norman Steiner of ACE IT Solutions, operates out of Northern New Jersey and New York City, two of the regions that were the most devastated by the storm. Norman commented on his Sandy recovery experience with Datto: “Using [Datto’s] SIRIS product, we were able to bring up a Hedge Fund Client in Lower Manhattan on Monday the 29th of October in your cloud. By Tuesday [Datto’s] team had created 15 VPN logins into the Datto cloud which we were able to give out to their key employees for access. Timing was critical since the Stock Market reopened on Wednesday and the client needed to access data from their server beforehand. Thanks to Datto, we were able to provide the Disaster Recovery we promised to our client when we sold the SIRIS device originally. We’d like to thank Datto for providing a product that actually works.”
Another Datto partner, Manny Mamakas of Greenwire Solutions in Scarsdale, New York, had a non-profit client in New York that assists adults with special needs. Being located just hundreds of feet from the water on Staten Island, the client’s server room was under 5 feet of water due to the storm surge caused by Sandy. Their VMWare server and other network equipment was destroyed. Manny was able to instantly virtualize his client’s server through Datto’s portal and after coordinating with Datto’s technical support to NAT the VMs to a public IP address, the server was ready for production. The next morning Manny’s client was able to run their business successfully in the cloud while their servers still sat underwater. The most gratifying part of Manny’s story is that his client had started using Datto’s SIRIS only two weeks prior to the storm.
Some Datto employees manned shifts for as long as 24 hours and stayed at Datto’s HQ for days on end to assist customers in bringing up virtual machines. Even members of tech support who had only been on the job for six weeks were sleeping at their desks so they would be ready to receive support requests immediately. The level of commitment was unparalleled—technical support did not miss a single SLA goal for the duration of the storm and the days that followed. This commitment was “a testament to the unwavering dedication of Datto employees to assist our partners through this disaster,” said Datto CEO, Austin McChord.
With the worst behind them, the Datto team continued to maintain 24-hour staffing of both their support department and their Cloud Operations Center to support the thousands of the devices that were affected by the storm. After the storm, Datto was effectively supporting 100 times their monthly average of disaster scenarios in their off-site cloud. Although partners were virtualized in the cloud, Datto still had to prepare for their partners’ next steps toward full recovery—the physical restoration of their servers and production machines in their local environments.
When and if another super storm hits, make sure you are prepared and your data is in the right hands.