Out of the Ashes: How AirCon Technologies
Survived the Fort McMurray Fire

1.2M Acres
2,400 Buildings
80,000 Displaced
$9B in Damages

In May 2016, the city of Fort McMurray, Alberta suffered a devastating wildfire. The fire burned more than 1.2 million acres, destroyed 2,400 buildings, and displaced more than history. Damages have been estimated to be in excess of $9 billion.

Entire sections of the city were completely destroyed, and many businesses in the region were severely impacted by the event. Many suffered extended periods of downtime.

AirCon Technologies was one of the many businesses operating in the evacuation zone. Though AirCon employees wouldn’t return to Fort McMurray for over 40 days, they were able to return to operations quickly thanks to smart decisions, the right technology, good fortune, and a lot of hard work. This is their story

Tue, May 3, 2016
8:00 AM

Liam Burgess, President and CEO of AirCon Technologies, was on his way to the company’s Fort McMurray office. The previous afternoon, there had been black smoke visible from a wildfire burning in the forest southwest of the city, but Burgess didn’t think much of it. “It’s a very wooded area, so fires are somewhat common. I didn’t even pay attention and figured it would be no big deal,” he said.

He was not alone. For residents of the city, it was pretty much business as usual. And, in Fort McMurray, business is oil. Located in the middle of Northeast Alberta’s Athabasca oil sands, it is the heart of one of Canada’s largest hubs for oil production. According to Burgess, almost everyone in the area is in some way connected to the oil industry.

AirCon Technologies is no exception, the company installs and services fire suppression, air conditioning, heating, and air filtration systems on a broad range of mobile heavy equipment including some of the largest hydraulic shovels and haul trucks in the world. Its clients include some of the largest oil production companies operating in Western Canada today.

The morning was like any other at AirCon, except that Burgess was in a series of meetings about a large contract that the business had just landed. Outside the sky was clear and there was no sign that the fire was becoming a threat. In reality, however, this couldn’t have been further from the truth.

The previous afternoon, the fire had spread to about 1,250 acres, according to The Globe and Mail. There were 80 firefighters at the scene, attacking it from the ground and by air. In spite of their efforts, the wildfire had more than doubled in size by 5:00 PM. By 8:00 PM, it had doubled again.

Hot, dry conditions and the vegetation in the area allowed the fire to grow quickly. Meanwhile, a weather condition called an “inversion” was at play, stoking the flames and keeping smoke low to the ground. On Tuesday morning, while Burgess and his colleagues worked, the inversion reversed, strong winds changed direction and the fire began tearing through the dense boreal forest toward Fort McMurray.

12:45 PM

“We were in a conference room and suddenly it got very dark, so I walked out see what was happening,” said Burgess. “I was immediately shocked by the smoke. It was blocking out the entire sky.”

According to Globe and Mail reports, two dangerous conditions known as “crowning” and “spotting” were occurring. Crowning happens when fire spreads rapidly through the tops of tall trees, independent of what’s happening on the ground. Crown fires can be particularly dangerous for firefighters, who must pull back and combat it from above using aircraft. Spotting occurs when embers from the fire are carried by the wind for longer distances—across a river, for example, which would normally form a natural fire barrier. By 1PM, the fire had jumped the Athabasca River and was threatening neighborhoods on the west side of Fort McMurray.

“Right away we sent everyone home,” said Burgess. “It was suddenly obvious that the situation was very bad.” At this point, neighborhoods in the south western part of Fort McMurray, such as Gregoire, Beaconhill and Abasand, were already on mandatory evacuation as a wall of smoke and flames stretched above the surrounding forest. In downtown Fort McMurray, AirCon employees started clearing out.

I was immediately shocked by the smoke. It was blocking out the entire sky.

3:00 PM

“It seemed like a typical day until the early afternoon,” said Calgary based Mark Cordingley, VP of Information Technology for AirCon. Around 2:00 PM, unaware of the events unfolding in Fort McMurray, Cordingley noticed something strange during routine monitoring of AirCon’s network—a rapid drop off of wireless network traffic. “I thought that was pretty odd, considering it was early in the afternoon, so I called the office,” he said.

In Fort McMurray, AirCon trainer and quality assurance manager Jacques Landry was the last employee in the office. He was still considering whether he would evacuate north or south of the city. When he picked up the phone, Cordingley asked him what was happening. “I’m the only one left,” he said. “Everyone else has evacuated.”

By this time, conditions were growing dramatically worse south and west of Fort McMurray. Strong winds continued to push the wildfire east, toward Route 63. The highway bisects Fort McMurray north to south and is the only route in and out of the city. According to Burgess, by 2:30 PM, traffic on 63 South was already extremely heavy with residents fleeing the city.

"Fort McMurray Wildfire 1194“ by Premier of Alberta, used under CC BY / Cropped from original

Meanwhile, as Cordingley and Landry sat on the phone watching the news coverage of the fire, it became clear that Landry needed to evacuate AirCon’s office. The fire had jumped another river, the Hangingstone, and spread into the residential neighborhoods of Abasand Heights and Beacon Hill, just to the west of downtown.

The two hatched a plan that Cordingley hoped would allow AirCon to restore business operations quickly with no data loss following the fire. Given the severity of the fire, they decided it was very likely the office and servers would suffer damages. At the very least, it seemed unlikely that they would be able to return to the office right away.

AirCon had a Datto SB2000 on premises, which backs up data continuously throughout the day and replicates to the Datto cloud daily at 10 PM. In the event of an outage, users can run applications from local or cloud backups while primary systems are being restored, allowing the company to avoid major downtime. Cordingley remotely logged into the system and saw that the last backup completed successfully at 3:02 PM, less than an hour earlier. He had Landry shut the system down and remove it from AirCon’s network.

Landry decided he would evacuate the city to the south with the SB2000. Once safe he would later hand it off to Cordingley, so he could restore data quickly in an alternate location if necessary. They took action just in time. Within an hour, Cordingley had lost all remote connectivity with the AirCon network due to the fire.

…

4:00 PM

Landry put the Datto device in his truck and headed south, but he didn’t make it far. The fire was spreading across Route 63, cutting off one of the two routes out of the city. He was forced to drive north, where many Fort McMurray residents would eventually find shelter at oil company work camps. By 4:00 PM, Route 63 North was a parking lot

"Landscape view of wildfire near Highway 63 in south Fort McMurray" by Darren RD, used under CC BY-SA 4.0 / Cropped from original

Around this time, Cordingley called Burgess, who was at home with his wife packing essentials and preparing to evacuate. “It was a really quick call,” he said. “I told him we have a live copy [of data and applications] as of 10:06 PM on May 2 and I’m working on May 3.” Burgess said that the call was a relief, because he had so much else going on. “My wife and I packed a bag and grabbed the dog and evacuated. Mark had our IT covered, that allowed me to focus on other aspects of the business.”

After he hung up with Burgess, Cordingley placed a support call to Datto. Even though Landry had taken the Datto device with him when he left the office, Cordingley wanted to understand all of his options. Given the deteriorating situation up north, he worried it could be a while before he could get his hands on AirCon’s SB2000. “When I called Datto, I got someone instantaneously,” said Cordingley.

Cordingley spoke with Michael Rollman, a Datto tech support engineer. Because of the unique circumstances, Rollman recommended an emergency reverse RoundTrip, where business data from the Datto Cloud is copied onto a hard drive and shipped to a partner, as well as setting up an FTP server so Cordingley could pull the data that way. “The situation with the fire sounded pretty chaotic, so we wanted to give them options,” said Rollman.

Rollman was working on Datto’s Code Red support team at the time of the fire. The rotating team is tasked with supporting partner’s disaster recovery efforts. Zacary Shannon, who heads up the Code Red team, said it was launched in February, to give partners enhanced support when they need it most—during a disaster recovery event. “It’s a dedicated team that handles DRs from start to finish,” he said. “There are dedicated cloud and Level 3 techs on the team, so there is no need to wait for that level of expertise if it is necessary.”

Wed, May 4, 2016
9:00 AM

The following morning, Burgess was in Edmonton. “We stayed north for a few hours, but our friend owns a plane, so we were able to evacuate to our secondary residence,” he said. At home, he was strategizing all things AirCon. One of his primary concerns was being able to run payroll on time.

“I care deeply about the people that contribute to my business,” Burgess said. “Families were displaced, there were lot of unknowns and an amplified level of fear. The last thing we wanted to do was financially stress impacted employees.” Cordingley felt the gravity of the situation as well. “Payroll was coming up, and 100 families were in various states of anxiety,” he said. “The data became vital, because payroll was about to be vital to these families.”

In Edmonton, Burgess spoke with AirCon’s accounting firm, Kingston Ross Pasnak LLP (KRP). “They offered us temporary office space to work from to keep operations going, which was extremely generous, and a true reflection of their level of customer care.” said Burgess.

2:30 PM

According to Global News Canada, 1,600 structures had already been destroyed by Wednesday afternoon. Beacon Hill, a neighborhood just west of downtown, was hit particularly hard—70% of the homes there had burned to the ground. There were no reports of damage in the downtown neighborhood where AirCon’s office is located. However, according to Burgess, there was really no way to know exactly what was happening there. North of the city, roughly 10,000 people were posted up at oil camps and another 70,000 had fled into surrounding communities. Alberta declared a state of emergency.

"Fort McMurray wildfires 10425“ by Premier of Alberta, used under CC BY / Cropped from original

Landry was still north of Fort McMurray, unable to deliver the Datto device to Cordingley. However, he was able to find a secure location to connect the device to the Internet. This allowed Cordingley to access the SB2000 remotely and perform a “delta sync” from the device to the Datto Cloud, to create a live image of the company’s data as of 3:02 PM the previous day. “We used this updated image to run the company during the initial time period before we set up the emergency office in Edmonton,” he said. “Being able to run from the cloud was a huge benefit.”

“The fact that we could spin up a virtual machine in the cloud and run payroll was very important to me personally,” Burgess said. “It made a big difference for our employees, who were very anxious about their homes.” Before Cordingley came on board, AirCon took a more traditional backup approach. “They were taking a file-based backup once a day,” he said. “If we hadn’t had Datto, and had to rebuild from scratch, it would take three days, and only provide data from four days ago. I’d estimate the company would have lost $30,000 in revenue in the process.”

If we hadn’t had Datto, and had to rebuild from scratch, it would take three days, and only provide data from four days ago. I’d estimate the company would have lost $30,000 in revenue in the process.

Fri, May 5, 2016
12:00 PM

According to Global News Canada, by Friday, more than 21,000 acres had burned. There were 49 separate active fires and seven of them were considered to be “out of control.” However, the fire was beginning to move east of the city into the dense forest. The threat was far from over as firefighters continued to battle hot spots in and around the city, but the situation was improved, at least for the time being. A massive convoy of vehicles began transporting residents who had fled north into the oil fields back across the city to locations south of Fort McMurray.

"Fort McMurray wildfires 457“ by Premier of Alberta, used under CC BY / Cropped from original

Cordingley got to work on the IT piece at his office in Calgary. He had to set up everything— desktops, printing, ESX server, managed WiFi etc. AirCon employees would arrive in Edmonton soon, so he worked quickly to acquire the technology he’d need to set up the temporary office space.

“The emotional atmosphere was so intense,” Cordingley said. “Some AirCon employees already knew they had lost their homes, others had not yet received confirmation. I don’t know what I would have done if I had to be the bearer of bad news about IT.” Thankfully, he didn’t have to be.

Over the course of the next two weeks, AirCon would run operations from in the Datto Cloud while Cordingley set up the temporary office space and production ESX environment in Edmonton. AirCon employees arrived on May 17. “The IT staff from KRP was stunned that we got up in running in the time we did,” said Burgess.

Conclusion

AirCon would operate in Edmonton for over a month, while the fire continued to burn near Fort McMurray. Thankfully, the company’s office was not harmed, although buildings nearby suffered damage. AirCon returned to Fort McMurray on June 7, 2016.

“Mark is extremely dynamic, he enabled me to concentrate on other aspects of the business during that critical time,” said Burgess. “On the business side of things, we were able to show the strength of our operations by getting back online so quickly. Some of our competitors were unable to do so. As difficult as it was, it was a positive for the business, and Datto was a part of that.

Many businesses in the area did not fare as well. The Conference Board of Canada estimates that the fire resulted in a loss of 1.2 million barrels of oil per day for two weeks, which translates to nearly $1 billion.

Natural Disaster Survival Guide For Businesses

Free eBook

Natural Disaster Survival Guide For Businesses

A Quick Reference for Business Leaders

Download the eBook