October 18, 2017
Podcast: The Datto Disaster Recovery Team
In this episode of the datto podcast, I am joined by Matt Massuch, our director of technical support, and 2 members from our disaster recovery team, Justin Hunter and Steve Homick to discuss what happened on the ground during hurricanes Harvey and Irma. Check it out below!
Kevin Williams: Hello, and welcome to another episode of The Partner Technical Podcast. I'm your host Kevin Williams, one of the videographers here at Datto. Today I'm joined by our Technical Support Director Matt Massuch, and two of our Disaster Recovery Team members that went down to Texas and Florida, Justin Hunter and Steve Homick. Guys, thank you very much for joining me today.
Kevin Williams: On this episode, we wanted to go over the Disaster Response Teams, how they were created, and how they're able to assist. Matt, jumping right into it, a Field Response Team isn't something that we're used to seeing in our industry. What would you say drove us to make that decision overall?
Matt Massuch: I think it was just the response that we wanted to have for our partners that we knew were in affected areas. For those of us that were here during Sandy, obviously it was quite a storm up here. And this was a much larger magnitude of a storm. And we saw the effect it had with our partners, specifically in us being able to respond to their disaster recovery needs. I think this just ended up being a no-brainer at the end of the day. Let's put some people on the ground close to where they can assist.
And we didn't have our networking line back then, so we had a really nice opportunity to be able to provide partners the connectivity that they would need in the result of the storm, given the fact that the infrastructure would be pretty significantly damaged.
Kevin Williams: Now knowing that time is really not on anyone's side for either of these situations, how quickly would you say the team was able to be put together?
Matt Massuch: I think the first team took me about five minutes to put together. So it was Friday morning, we had started talking about this Thursday night. Friday morning about 7:55 in the morning, I literally walked right into work and asked people, "Who wants to drive down to Texas today?" And I got responses immediately from Justin, who was one of them. It was incredible. People were just on board right away, and they knew what we were trying to do, and they knew what we were trying to put together. And we had the car ready to go within about, what, four hours I think?
Justin Hunter: Something like that.
Matt Massuch: Yeah, four hours to get ready for the trip to Texas. So it was an incredible response from the team.
Kevin Williams: Now in comparison, adding both hurricanes together, how many teams would you guys say that you've sent down total?
So we sent down a total of six teams, three teams for Harvey and three teams for Irma. And the size of the team is about three to four people. Yeah, three to four people each.
Kevin Williams: Now what's the process for deploying a team? Obviously this has already happened twice within not even two months. So what's the process in the future for deploying a team like this?
Matt Massuch: So as far as the Disaster Recovery Team is concerned, it is something that we want to look to continue to stand up. So it's gonna be a combination of people that have done it before, people that had the experience, and there's the potential to put more training around the process. If you were to be deployed as part of this team, you have some additional training, additional know-how for when you're going into an unknown, when it comes to storms or any sort of related event, weather or what have you. To figure out who's the best team for it, who are the best team members to be able to do it, who has the experience.
So we're going to look at a lot of that, and try to figure out how we want to keep the cadence going in the future. So this is definitely something that we want to, we're going to continue to invest in. And it's something that we want to make sure is ready to go next time they're needed.
Kevin Williams: Now Justin, I understand that you kind of jumped into this. It wasn't like you were prepared for it. Day in, it was just like, "Hey, I'm gonna go to Texas today," that kind of a thing. Getting into the car, though, you guys didn't necessarily have a precise destination, especially with the ground conditions constantly changing. What was it like, from your team's perspective, to go into the storm like that?
Justin Hunter: So it was not really completely planned for the beginning exactly where we were headed. But as we drove down, that became more of a solid picture of what our actual destination was. We were pretty much just pointing towards Texas and where they thought the hurricane was going to go at that time. When we got into the van and we just started heading down there, there was a couple things that we used. Glympse was one of our biggest tools to at least show people where we're going. If you haven't used the app before, it's just a GPS tracking app that anybody can use a link, and see in live time where you're going, where you are on a map like a Google Map.
We used that, we put that up in the Slack room so that people who were planning it here were able to see where we were in live time, and where we were headed. And we kept in constant phone and text messaging communications with people like Matt Richards, who were here actually doing the planning, so that they could tell us what our actual final destination was.
Matt Massuch: And I think Austin actually put that Glympse app up on Twitter as well, so we could showcase the live tracking of us heading down to the storm.
Kevin Williams: You're literally watching them go the whole time.
Matt Massuch: Yep, whole time.
Kevin Williams: Now Justin, what was the thought process behind having the DNA in the van? I understand that you guys obviously stocked the car full of devices for partners, but for you guys to have one, what was the thought process behind that?
Justin Hunter: So the story with that was actually, we were driving the van. It was pretty early, and we were using the Glympse app because Sergio brought up the idea of using the Glympse app. And we were using the Glympse app, and it kept on losing connection. And we were trying to figure out, oh, we don't have any signal here, or whatever. And I don't remember, it was either Sergio or Rich were like, "Well we have 30 DNAs in the back. Why don't we just grab one of those and hook it up? And then we'll have constant internet connection."
So that's what we did. We stopped at a rest stop on the side of the road, and we spent about two minutes getting it all connected. Sergio was able to do both sides of the DNA connection process, so we were able to get it up and connected to the reseller that he created. And once it was like that, we had interaction for the entire 30-some odd hours that we were driving down. So it made keeping in contact a lot easier than before we had that in the van.
Kevin Williams:So you've got the communication going back and forth, you've got the app where we're able to really track your location. You finally get down there. What's it like when you first get down there?
Justin Hunter: So the plan when we first got down there was kind of keep our heads low at first, because the hurricane was still there. So I remember the first morning that we woke up and we walked outside, the wind and rain was literally sideways. And you stood underneath the covered area in front of the hotel, and you were still getting pelted with rain. You had to stand in back of a car to not be able to get hit. The weather was pretty insane.
But again, we were in the planning process. We were still trying to figure out exactly what the team was supposed to be doing. So I believe the first or second day, we had a partner to go to. ICS was first, and they were about five minutes away from the hotel. So it was go there, and then sit down, and talk as a team, and figure out what partners need, how to reach out to partners. Basically what's going on? What can we do while we're here? Because we know we have partners that are in the area, we know we have partners that are in business trouble. What can we offer them? What can we do for them?
So that was pretty much what was going on. And the first day or two, it wasn't as busy as we thought it was gonna be. But we came to the conclusion that that was because people were putting personal and property before business. So they weren't so much interested in grabbing a DNA for business, because they were interested in, "Hey, my house is flooded."
Kevin Williams: Right, I understand that. What kind of equipment, in addition to DNAs, did you guys bring down with you?
Justin Hunter: So we had the DNAs, we had the switches, we had the AP60s, we had some SX40s. We just had anything that they had at build, that they thought we may have needed while we were down there. They just threw at least a couple in just in case we needed it. We had the Surge Three Image Sticks. Yeah, it was pretty much ... The process was we were getting ready here, Matt came up and offered us here. And while that was going on, Kevin White and Zach were getting the van ready, and figuring out what we needed to bring. And they just loaded the van with whatever they thought, including safety equipment. We had a generator, I mean that kind of stuff.
Kevin Williams: Once things started to settle as far as directions to what you needed to do, like, "All right, we have a partner we can go with," or I know you mentioned ICS was one of the partners that ended up helping you guys out. Or you were able to stay with them to assist.
Justin Hunter: Yeah, that's who we first stayed with. They're in Austin. Like I said, they were around the corner, which the roads in Austin ... They were five minutes from the hotel, and then they were like 12 minutes back to the hotel. That was just how the roads work in Austin, but they were right around the corner. And they were helpful. They literally, "Hey, do you want us to order you lunch? Hey, do you need help with something?" They were extremely helpful.
Kevin Williams: Were there other partners within the area that were offering those same kind of ... Like, "Hey, you're here to help. Please come here." Were there any other partners that were similar to ICS?
Justin Hunter: There were. One of the first partners, I apologize, I don't remember what the partner's name was. But one of the first partners that came to get a DNA, they told us that, "If you need any help, if you need anything from a partner standpoint, just let us know." So a lot of the partners that we met with were, again, very helpful and very appreciative that we were there doing what we were doing. So they wanted to also lend a hand if we needed it.
Steve Homick: The outreach was incredible that we did to our partners. And their response was really positive that we were coming down to be able to assist them. So it was a really good effort all the way around.
Justin Hunter: The thing we kept on hearing was, "You know no other company would do this, right?" We just kept on being told that over and over again. So that was pretty amazing.
Kevin Williams: You're camping out here in this mobile command center of sorts. What were some of the things you guys were able to do from a support standpoint, now that you're kind of set up and no more traveling at this point? You've set up camp here.
Justin Hunter: Our biggest thing was figuring out how to set DNAs up for the partners, because it was a different process. Because usually a partner calls in, orders a DNA, it's attached to the invoice, the invoice basically helps attach it to the partner, and then the partner gets shipped the DNA. We're kind of doing this in the complete opposite direction. So we're handing them the DNA that had no connection to it.
So we had to manually connect it to their partner, make sure it's working, and all of that. And a lot of the DNAs that we were handing out, even during Irma, because of the demand, were refurbs. So a couple of the times, we had to make sure they were working, get on, and then wipe all of the stuff because there were already configurations on the DNAs. So there were a couple steps that we had to go through before we could hand it to the partner.
But one of the biggest things we tried to do, and what I was doing when we were setting up in Irma especially, was when a partner came to get a DNA, I said, "Hey, have you ever set this up before?" And they said, "No, I have nothing to do with DNAs." And I said, "Okay, let me take a second and show you. This is a configuration page. This is how you do this." Because a lot of the partners hadn't touched a DNA before. We were offering these DNAs, but they had no real experience with the product at all.
Kevin Williams: With Irma being as widespread as it was, in your opinion, what were some of the biggest differences you saw from Harvey to Irma? I know that you had obviously gone to both storms. Fro
m your perspective, what were some of the bigger differences?
Justin Hunter: Harvey was a lot more centralized, as far as where it was in Houston. When we first went to Austin, because Austin is where we first started with ICS, there wasn't too much around there that was affected. A lot of the partners that we were getting calls from were partners that were in or around Houston. And a lot of the questions were, "Hey, when are you coming to Houston? Because we need you here." And a lot of the partners that we handed DNAs to were ones that made the hour and a half, two hour drive to come get a DNA and then go back.
The difference with Irma is Irma went right up the coast, and wiped right across Florida. So it affected such a widespread area that we had people calling from like five hours away in Miami, or whatever, and saying, "Hey, I need a DNA. When are you coming down here?" And it's just one team. It was hard for us to say ...
Kevin Williams: Go widespread.
Justin Hunter: Right. It was a lot harder, but it was a lot busier at first. Because there were a lot of areas that were affected with just no power. And the partners' houses or their personal property weren't affected. So they were immediately interested in taking care of business. So within the first day, I don't know if you remember the numbers, Steve, we had like 15 or 16 DNAs that went out the first day.
Steve Homick: At least.
Justin Hunter: Which was totally different, because the first two days in Harvey, we had maybe three. And we brought 32.
Kevin Williams: Big difference there.
Justin Hunter: Yeah.
Kevin Williams: Now Steve, from your perspective, I know that you were on one of the first teams to go down to Irma. You're seeing a lot of these team members from Harvey come back, and you're hearing their experiences and overall what they learned. What were some of the biggest things that your team brought down to Irma, learning from what had happened in Harvey?
Steve Homick: I mean, seeing the response and the way they acted so quickly, and kind of having a limited tool set, I guess, just simply because of time not being of the essence with getting deployed for Harvey so quickly, I sort of felt grateful to be on the team that we had a little more time to think, a little more time to get things going equipment-wise. I mean we had an awesome, big truck pretty much suited for hurricane activity going down there. And we sort of knew what we were going into. Geographically, it affected a different widespread area, with Florida being a state with counties wedged closer together than the great big state of Texas, where everything was a little more central to Houston.
I guess we had more time to prepare for Irma Team One than the guys at Harvey did.
Kevin Williams: That's understandable. That's totally understandable. If anything, guys, what were some of the biggest takeaways that you all learned from these experiences? Whether it's going down there, Matt, from your perspective, literally keeping everything organized here. What were really some of those big takeaways for you guys?
Matt Massuch: For me, I think a lot of it was the response from the support team, and the company in general. That we had this idea on Thursday night, right before the storm. Let's get everybody going to it. Everybody got behind it, everybody was bought into it. And this was coming all the way from Austin. Because basically when we presented the original idea, he said, "Do it. You have my approval." And we just put it in motion.
And just the entire response of the company, to get behind this effort, I think probably was one of the most incredible things that I've ever seen. It really was.
Kevin Williams: That's great.
Steve Homick: Talking to partners every day, hearing what they're going through with clients, having the need to get servers going up; it's one thing to hear them and work on tickets and all that. But to see people come to a hotel at 11:30 at night to get a DNA, to get their clients going again. You feel that sense of urgency. And you could almost feel that since we were there physically, though, we were there for them in a physical response presence during these natural disasters. So I think it formed our partnerships stronger doing that.
Justin Hunter: Yeah. Speaking of the partner point, one of the biggest shout-outs that we want to make, as far as our partners are concerned, is ... I forget which one, ICS or ... I think it was ICS. Mike [phonetic: Chassam 00:16:40], the partner who was in Houston with us. When we first got there, he wasn't just, "What do you guys need?" He literally sat in the boardroom with us, called shelters, he was so hands-on and so helpful with us. It was just incredible to see from one of our partners who wasn't necessarily getting paid to do that by us. He was just ready to volunteer, and do whatever was needed. I mean, it was just amazing to see, to actually interact with partners that you usually only get to hear on the phone.
Kevin Williams: Big difference from that perspective. Guys, thank you, all of you, for being on the podcast today. We really appreciate all your insights, and getting to see what really, what it took to get this all together. Everyone, please stay tuned for more technical tips and tricks.