Tue, May 3, 2016
Liam Burgess, President and CEO of AirCon Technologies, was on his way to the company’s Fort McMurray office. The previous afternoon, there had been black smoke visible from a wildfire burning in the forest southwest of the city, but Burgess didn’t think much of it. “It’s a very wooded area, so fires are somewhat common. I didn’t even pay attention and figured it would be no big deal,” he said.
He was not alone. For residents of the city, it was pretty much business as usual. And, in Fort McMurray, business is oil. Located in the middle of Northeast Alberta’s Athabasca oil sands, it is the heart of one of Canada’s largest hubs for oil production. According to Burgess, almost everyone in the area is in some way connected to the oil industry.
AirCon Technologies is no exception, the company installs and services fire suppression, air conditioning, heating, and air filtration systems on a broad range of mobile heavy equipment including some of the largest hydraulic shovels and haul trucks in the world. Its clients include some of the largest oil production companies operating in Western Canada today.
The morning was like any other at AirCon, except that Burgess was in a series of meetings about a large contract that the business had just landed. Outside the sky was clear and there was no sign that the fire was becoming a threat. In reality, however, this couldn’t have been further from the truth.
The previous afternoon, the fire had spread to about 1,250 acres, according to The Globe and Mail. There were 80 firefighters at the scene, attacking it from the ground and by air. In spite of their efforts, the wildfire had more than doubled in size by 5:00 PM. By 8:00 PM, it had doubled again.
Hot, dry conditions and the vegetation in the area allowed the fire to grow quickly. Meanwhile, a weather condition called an “inversion” was at play, stoking the flames and keeping smoke low to the ground. On Tuesday morning, while Burgess and his colleagues worked, the inversion reversed, strong winds changed direction and the fire began tearing through the dense boreal forest toward Fort McMurray.
“We were in a conference room and suddenly it got very dark, so I walked out see what was happening,” said Burgess. “I was immediately shocked by the smoke. It was blocking out the entire sky.”
According to Globe and Mail reports, two dangerous conditions known as “crowning” and “spotting” were occurring. Crowning happens when fire spreads rapidly through the tops of tall trees, independent of what’s happening on the ground. Crown fires can be particularly dangerous for firefighters, who must pull back and combat it from above using aircraft. Spotting occurs when embers from the fire are carried by the wind for longer distances—across a river, for example, which would normally form a natural fire barrier. By 1PM, the fire had jumped the Athabasca River and was threatening neighborhoods on the west side of Fort McMurray.
“Right away we sent everyone home,” said Burgess. “It was suddenly obvious that the situation was very bad.” At this point, neighborhoods in the south western part of Fort McMurray, such as Gregoire, Beaconhill and Abasand, were already on mandatory evacuation as a wall of smoke and flames stretched above the surrounding forest. In downtown Fort McMurray, AirCon employees started clearing out.I was immediately shocked by the smoke. It was blocking out the entire sky.
Landry put the Datto device in his truck and headed south, but he didn’t make it far. The fire was spreading across Route 63, cutting off one of the two routes out of the city. He was forced to drive north, where many Fort McMurray residents would eventually find shelter at oil company work camps. By 4:00 PM, Route 63 North was a parking lot
Around this time, Cordingley called Burgess, who was at home with his wife packing essentials and preparing to evacuate. “It was a really quick call,” he said. “I told him we have a live copy [of data and applications] as of 10:06 PM on May 2 and I’m working on May 3.” Burgess said that the call was a relief, because he had so much else going on. “My wife and I packed a bag and grabbed the dog and evacuated. Mark had our IT covered, that allowed me to focus on other aspects of the business.”
After he hung up with Burgess, Cordingley placed a support call to Datto. Even though Landry had taken the Datto device with him when he left the office, Cordingley wanted to understand all of his options. Given the deteriorating situation up north, he worried it could be a while before he could get his hands on AirCon’s SB2000. “When I called Datto, I got someone instantaneously,” said Cordingley.
Cordingley spoke with Michael Rollman, a Datto tech support engineer. Because of the unique circumstances, Rollman recommended an emergency reverse RoundTrip, where business data from the Datto Cloud is copied onto a hard drive and shipped to a partner, as well as setting up an FTP server so Cordingley could pull the data that way. “The situation with the fire sounded pretty chaotic, so we wanted to give them options,” said Rollman.
Rollman was working on Datto’s Code Red support team at the time of the fire. The rotating team is tasked with supporting partner’s disaster recovery efforts. Zacary Shannon, who heads up the Code Red team, said it was launched in February, to give partners enhanced support when they need it most—during a disaster recovery event. “It’s a dedicated team that handles DRs from start to finish,” he said. “There are dedicated cloud and Level 3 techs on the team, so there is no need to wait for that level of expertise if it is necessary.”
Wed, May 4, 2016
The following morning, Burgess was in Edmonton. “We stayed north for a few hours, but our friend owns a plane, so we were able to evacuate to our secondary residence,” he said. At home, he was strategizing all things AirCon. One of his primary concerns was being able to run payroll on time.
“I care deeply about the people that contribute to my business,” Burgess said. “Families were displaced, there were lot of unknowns and an amplified level of fear. The last thing we wanted to do was financially stress impacted employees.” Cordingley felt the gravity of the situation as well. “Payroll was coming up, and 100 families were in various states of anxiety,” he said. “The data became vital, because payroll was about to be vital to these families.”
In Edmonton, Burgess spoke with AirCon’s accounting firm, Kingston Ross Pasnak LLP (KRP). “They offered us temporary office space to work from to keep operations going, which was extremely generous, and a true reflection of their level of customer care.” said Burgess.
According to Global News Canada, 1,600 structures had already been destroyed by Wednesday afternoon. Beacon Hill, a neighborhood just west of downtown, was hit particularly hard—70% of the homes there had burned to the ground. There were no reports of damage in the downtown neighborhood where AirCon’s office is located. However, according to Burgess, there was really no way to know exactly what was happening there. North of the city, roughly 10,000 people were posted up at oil camps and another 70,000 had fled into surrounding communities. Alberta declared a state of emergency.
Landry was still north of Fort McMurray, unable to deliver the Datto device to Cordingley. However, he was able to find a secure location to connect the device to the Internet. This allowed Cordingley to access the SB2000 remotely and perform a “delta sync” from the device to the Datto Cloud, to create a live image of the company’s data as of 3:02 PM the previous day. “We used this updated image to run the company during the initial time period before we set up the emergency office in Edmonton,” he said. “Being able to run from the cloud was a huge benefit.”
“The fact that we could spin up a virtual machine in the cloud and run payroll was very important to me personally,” Burgess said. “It made a big difference for our employees, who were very anxious about their homes.” Before Cordingley came on board, AirCon took a more traditional backup approach. “They were taking a file-based backup once a day,” he said. “If we hadn’t had Datto, and had to rebuild from scratch, it would take three days, and only provide data from four days ago. I’d estimate the company would have lost $30,000 in revenue in the process.”If we hadn’t had Datto, and had to rebuild from scratch, it would take three days, and only provide data from four days ago. I’d estimate the company would have lost $30,000 in revenue in the process.
Fri, May 5, 2016
According to Global News Canada, by Friday, more than 21,000 acres had burned. There were 49 separate active fires and seven of them were considered to be “out of control.” However, the fire was beginning to move east of the city into the dense forest. The threat was far from over as firefighters continued to battle hot spots in and around the city, but the situation was improved, at least for the time being. A massive convoy of vehicles began transporting residents who had fled north into the oil fields back across the city to locations south of Fort McMurray.
Cordingley got to work on the IT piece at his office in Calgary. He had to set up everything— desktops, printing, ESX server, managed WiFi etc. AirCon employees would arrive in Edmonton soon, so he worked quickly to acquire the technology he’d need to set up the temporary office space.
“The emotional atmosphere was so intense,” Cordingley said. “Some AirCon employees already knew they had lost their homes, others had not yet received confirmation. I don’t know what I would have done if I had to be the bearer of bad news about IT.” Thankfully, he didn’t have to be.
Over the course of the next two weeks, AirCon would run operations from in the Datto Cloud while Cordingley set up the temporary office space and production ESX environment in Edmonton. AirCon employees arrived on May 17. “The IT staff from KRP was stunned that we got up in running in the time we did,” said Burgess.