How to Handle a "Worst Case Scenario" DR Situation

Most MSPs probably have not experienced a disaster recovery situation, and that’s a good thing. But, if the time comes, here are some best practices from Datto partner, Chad Kempt.

Rob Rae:

Hey, everybody. Welcome to this week's Datto Partner Podcast. Rob Rae here, Vice President of Business Development, happy again this week to be your host.

Today we've got one of our partners, an MSP of ours by the name of Chad Kempt. He's the founder of Fast Computers. He's here to discuss some recent situations that caused one of his clients, a small business, to actually have some downtime. Obviously, it's simply not acceptable. He had to do something about it. We actually went through the disaster recovery scenario. As a matter of fact, a lot of why we position and sell disaster recovery, business continuity, backup disaster recovery, a lot of what you guys are talking about, is those DR scenarios. When you actually do lose that company, you lose that data, you lose something along those lines where you actually have to go through the DR scenario.

We're going to walk through that process today, because as much as that's the core reason why we actually position and sell this technology, it's very rare where we have to do it, which is ultimately a good thing. However, what happens in those actual scenarios? Chad's got a couple of examples and some cool stories that we're going to talk about, but not to spoil that ending, Fast Computers has used Datto to save the day with virtualization and virtually no downtime as an actual result. We'll go through that as well.

Chad's also here to discuss a little bit about why Fast Computers uses Datto in the first place, how it compares to other build your own solutions and just how Datto and Fast Computers have been able to repair and restore and keep small business as well as other businesses up and running. First of all, let me introduce you to Chad. Chad, thank you very much for joining us today. If you want to jump in here and give us a quick introduction of yourself and a little bit about Fast Computers.

Chad Kempt:

Sure, Rob. Thanks for having me on today. Fast Computers started out as a one-man operation where I did a variety of consulting in '98. That's when we started. In 2007, I hired my first permanent full-time employee. He's still with me actually. In 2008, we opened a physical brick and mortar location where we're still currently located. We're in southern Ontario, just outside of the GTA. We cover Toronto to Niagara to London. It's our geographic area. If we had to roll trucks all day, every day, that probably wouldn't be so feasible, but due to the wonders of remote technology, we're able to extend our range a little bit.

We used to do a lot of what I call blue collar IT, which is warehouses and manufacturing, places where basically you show up and you need steel-toed boots instead of suits and ties. We really don't focus on one target vertical. We have a target client type in mind we like to work with and that is anybody that fits under a professional services umbrella. People who have desktops they sit at, servers they rely on, technology that runs their business, office type environments.

We've grown from what we were doing in the early 2000s to where we are now and we're actually looking at getting a new office space, possibly this year, as we've kind of outgrown our current one, unless somebody can tell me where they sell double-bunk desks, where I can stack up my staff. We're out of floor space. Early on, we would sell what I refer to now as the flavor of the week. That is any product or service that seemed like the best deal at the time or the new amazing thing to use. This didn't seem as terrible at the time as it does now, because it was just me dealing with it and I've always kept up on tech, personally outside of work.

When I started adding staff, the retraining that goes with new products and the confusion around who has what product became blatantly obvious. It was shortly after that that I started standardizing heavily on products and our service offerings across the business.

Rob Rae:

In essence, becoming a managed service provider, which is becoming the optimization of those products and services.

Chad Kempt:

Yeah. Exactly.

Rob Rae:

Maybe talk about, as an MSP, because I get a lot of these questions and when you take a look at how an MSP runs their business, it is many, many different flavors, and I'm sure you've seen this as well. Some of it is all-you-can-eat, some of it is they only focus on certain products and subsets. Some of them you can buy a la carte services. How does Fast Computers set that up and what are some of the vendors that you use in order to make up what your core MSP business is? Do you have a that you have to take it all or we won't do business with you kind of solution?

Chad Kempt:

That's a great question. That's a good question, Rob. I get asked that actually, too, by other companies when we're chatting. What I used to do ... I should say when we first got into managed services, we were basically throwing stuff at the wall and seeing what would stick. Obviously, that's not a good model, so what we ended up subscribing to was the everything or nothing, take it or leave it approach. We did that for a while and it was pretty successful, but the problem I was having was I don't like leaving money on the table. I don't want what we would all call bad clients. I don't want abusive clients, but I found that, if we say that 20% of the market is the target for, they'll take everything and it's great and they're the best.

Then say 60% of the market is good customers that pay their bills, understand what you're selling them. They want a good service, good products, but they may not need or want quite that, what do you want to call it, a Cadillac level of service. Then you get 20% of the market that I don't want. They can go to Craigslist. What I have done is we've created this hybrid model where we still do offer the package that suits the top portion, but we don't alienate the middle portion at 60%.

What we've done in the 60% portion and, in order to accommodate this, is we've broken our services up, rather than saying it's this much per user and if you don't pay that, then you're out of luck. We've actually gotten in with a lot of IT departments and a lot of clients where we wouldn't have been in otherwise by saying, "Well, what do you need? You need a backup and disaster recovery solution? Great, we have a Datto solution. Oh, you have advanced networking needs and your staff just doesn't have that skill set? No problem. We're a Fotonet partner and we can come in, we can set up your networking. If you want a little bit more than that, we can even manage your networking but, if not, that's fine too."

We've created these sort of pockets of offerings that we're very good at, software development and other things, and we're able to uniquely position these things that we're very good at and we've even gotten in with other IT companies. It's actually been a really good process.

Back to what I was saying about the all-you-can-eat thing, like I said, that works good for some people, but we have some customers where another example is we're a Lexmark dealer. I'm not rolling trucks all day fixing printers, but we'll go do an install and the customer will buy consumables from us for the next few years. There's a lot of revenue there. I don't like leaving money on the table.

Rob Rae:

I know you get an opportunity to travel quite a bit for work. I find that a lot of our more successful MSPs are the guys that actually go out and spend a few weeks a year going out to trade shows, going out to looking for what's new, looking for new vendors, also that networking with other MSPs is also really, really critical and we'll talk a little bit about that or comment on that at the end of this. I know I've seen you traveling quite a bit and engaging with other MSPs and, being Canadian, do you see any real fundamental difference between a Canadian MSP versus what an American MSP would typically deal with?

Chad Kempt:

That's actually a great question. When I'm talking to ... We'll say more on a typical level, so a typical American MSP seems to have different ... The customer mindset in the US seems to be slightly different. The Canadian mindset is a lot more guarded. They tend to be a lot more conservative with their spending and so when I talk to Canadian MSPs, a lot of them have a lot of difficulty getting traction with some of the product offerings. It's too expensive or this or that. The thing with that is, it comes down to trust and when we're working with new clients or existing clients, it's all about building a relationship with them and, if you can't build that relationship, it's not going to matter.

We've taken over a lot of clients that had very, very, very poor infrastructure prior to us. I'm talking ones ... We've taken over some where they told me they don't have backups and then they gave me a reason that I can't even ... I wasn't even able to process. I'm not talking bad backups. I'm talking no backups. For example, the last guy sold us a RAID in our server, so we don't need backups. Just insane things like that. It's a big step to go from somebody that's running a bunch of refurbished computers and a server that doesn't need backups because it has "RAID", to telling them they need to buy proper server infrastructure with proper warranties and proper networking equipment. The thing that they got down at the local box store for 50 bucks isn't going to cut running their office of 25 employees.

There is some challenges there, but you have to work through them and the customer oftentimes was never offered these things in the past. Again, that's what I see up here. I'm not saying that never happens down in the US and the US is a very large country, so there's definitely pockets everywhere. When I talk to people from LA, I get a much different vibe than when I talk to people from New York and when I talk to the Midwest, it seems to be very specific by the area what the attitude is.

Rob Rae:

You would suggest that it's not easier in Canada than it is in the United States?

Chad Kempt:

Oh, no, I would actually argue it's the other way, just from talking to other MSPs up here and my own experience.

Rob Rae:

Fair enough. Let's talk a little bit about Datto. Obviously, you went through this process where you go through and select the different vendors that you work with. I think every MSP or reseller back in the day starts with this, whatever the customer's got, we'll bring it on board, we'll work it and service it. Demand in services market has a tendency to more default towards standardization of specific technologies. In this particular case, you standardized on Datto, which is fantastic and we appreciate that.

Maybe walk us through the early days when you first took a look at us and what was the value at? What was the difference? What was the nuance that made you say, "I want to put this into my managed services offering and offer that to my customers?"

Chad Kempt:

Well, I can't really get to that without mentioning what we were doing before and what brought us to that. What we were originally doing before was we were doing a build-your-own-nightmare solution and what we were doing with that was we were standardized on the equipment we were using and the software we were using and everything. The problem was it wasn't very consistent. Prices would be changing all over the place. It was difficult to lock down consistent costs. The interface wasn't very junior tech friendly. Even for things like flower stores, we were involving more senior techs. All these sorts of things added up to culminate into giving Datto a look.

At first glance, it seems like it's a lot more expensive than what we were dealing with, but the problem is, we weren't measuring it correctly. What I mean by that is, our time has a value and especially when you can't utilize certain resources effectively, there's a cost to that. Actually, it turns out that Datto was actually quite cheap in comparison, because we weren't, inexpensive if you will, because we weren't dealing with that. When we first started with Datto, one of the challenges we had was that we were still dealing with a lot of the all-you-can-eat or nothing approach, and so we had these existing contracts that were built around pricing from our previous model.

What we ended up doing was those customers, we ended up putting in ALTOs for a lot of them because it was more in line with our cost structure. Back then, we were including everything. We were including firewalls. We were including a backup appliance. We were including obviously all the software and tools we needed to do our job. We ran into a situation with a client that was exiting and we had all their firewalls for all their locations for ours. What happened was essentially we had to go pick them up one day. Although it was in writing, there was a contract, we explained it, I don't think they fully understood the scope of that.

When we showed up to take their firewalls, it wasn't the best situation. Everything was by the book, so it wasn't a problem, but it was a little awkward. From then on, I decided, you know what, I don't want to have that situation again. There's no need for it. What we did was, instead of including a Datto as part of our offering, we started selling them. What we do is we'd say, "Here's the Datto. This is what it costs. It's going to cost this much per month or per year, depending on the customer, and if you don't like us anymore and you would like to fire us, that is not a problem. All you have to do is call Datto, tell them that you don't like us anymore and we'll be required to transition you over to a new partner and, to be frank, I don't want to work with anybody who doesn't want to work with us, so it's not a problem at all."

That's closed deals. You know what, I haven't lost a single customer from that. I think being able to do that and being very honest with them has helped us close more deals. It sounds counterintuitive, but it's been great. On that note, that packaging and that pricing has worked a lot better for us.

Rob Rae:

Let's talk about the actual recovery scenario. We do talk about the disasters. We talk about what causes the disasters. We talk about those different disaster scenarios and all MSPs I think have got several examples of how these things actually happen and rare scenarios come up where we have to do the recovery with them. Recovery, in essence, is what you're selling and positioning to the customer here, is what happens when. You've had a recent experience where you've had to do a recovery. It's an interesting story. Interesting enough that we wanted to do a podcast here, so that's the why we're here today. I'm really ... You could tell the story here for the crowd. Let's talk about the recovery. Let's talk about where it started, how it started, and then the process that you had and at what point were you sitting there on your knees with your fingers crossed going, "Please God, I hope this works."

Chad Kempt:

Actually, yeah. Let me just take you right through it. It really was a worst case scenario for us and for the client. We had started back the first day after the New Year holidays and there was an alert that there was a degrading status on the array. This was an inherited server, so it wasn't one we deployed and so there were some odd things about it we didn't like, but we were living with. The customer had been really oversold just before we came on board and so, although we did replace the backups obviously and the firewalls, we left that sort of stuff in place.

What ended up happening was the server had an audible beep on it and we couldn't turn off the audible beep without rebooting the server and going into the RAID bias, again because of the configuration of the server, and I asked the client if he could hold off until outside of hours, because it really was a bad idea to do this during business hours. He said that he could not wait. He would not remain sane until the end of the day.

We sent in a tech and when he rebooted to turn off the audible alert, it reported a multiple-drive failure. I think, at that point, it was two drives that had failed. It was RAID six, so it should have been able to handle a two-drive failure, but I guess a third drive actually failed during the process. We basically were down. The tech, he tried to ... To say he tried to get the array back online is wishful thinking is what he was doing. He probably spent two minutes doing some wishful thinking and when that was failing out, we had him move forward.

You have to understand that this was ... I wouldn't say the middle of the first day back, but I think it was around 11 a.m. or something like that, they have a main office where they have their exchange server and their terminal server, remote desktop server. Their second office relies on that office because their accounting software is there and without it up, they can't check customer balances. They can't print checks. They can't make invoices. They basically can't do anything. They can do work, but they can't bill anybody and they have no idea what anybody owes so this was horrible.

That's when we decided to abandon the old server and [inaudible 00:17:08] to Datto. We booted up the two VMs on the Datto, logged in and changed the IP addresses and he was back in business. We took the old server back to our office and, for anyone listening that's interested, when we were wiping the disks, testing them, five out of six of the drives failed during that time. Pretty much all the drives were toast. I should also note the customer was doing an old version of VMWare but, on the new server, we wanted to go to hyperV for various reasons, doesn't really matter. With the Datto backups, we were able to just copy the VHDs over to the new server after hours and boot them up. No issue. No conversion software. No extra work. No downtime. It doesn't get any better than that.

Rob Rae:

That's awesome, Chad. I want to thank you for sharing the story. I want to thank you for coming on and sharing your experience just as far as growing your business, what it is that you're working on. There's no doubt that you are one of the faster-growing companies that we've seen and that we continue to work with, obviously, thousands and thousands of MSPs, so it's nice when we get a story from a different part of the country, different part of the world, that kind of idea, to come in here and share with us. I really appreciate your time and appreciate you coming in and doing that with us.

Chad Kempt:

Thanks for having me.

Rob Rae:

Actually, Chad is going to be coming down to Norwalk, Connecticut the next couple of weeks. We're going to be recording a video together, which is going to be used for the video testimonial as well, so you might have an opportunity to see Chad in person. At the same time, we also have Chad and 700 other partners coming down to DattoCon which is going to be in June, June 20th through the 22nd, specifically down in Nashville, Tennessee.

We're also looking for content. If you have any interesting stories, any interesting angles, if there's anything you'd like to share or anything that you'd like to see during this podcast, we're more than happy to actually find someone who can actually perform that content for you. Don't hesitate to reach out with some feedback. Love to hear from you guys. Podcast again is a way to do that.

On behalf of Chad and everybody else here at Datto and myself, I want to thank Chad again for joining us, but I also want to thank you for taking time to listen to this. Stay tuned for next week's podcast and thanks again from Datto again. Take care.