July 2, 2015
Dear Silent Break Security:
My name is Robert Gibbons and I’m the CTO at Datto. I’m aware of your blog post discussing the security of Datto devices. I’d like to respond to the points raised in the post as well as offer some guidance that would have likely resulted in a better outcome for this device.
Before getting into the weeds, I first want to say that I truly appreciate the effort put into this post. I’m aware of the stereotype of a technology company digging in and resisting feedback on its products (e.g., shoot the messenger). Datto is not that. We welcome input because it makes us better. We realize that our products protect businesses’ most important asset - their data - and we are in a trust position with those we serve.
I also appreciate that the spirit in which you posted was to make all of the Internet, and Datto devices in particular, more secure by exposing these issues. While I don’t agree with all of your conclusions, you’ve uncovered some real vulnerabilities and we are taking immediate action to address them.
The rest of this response attempts to detail precisely where we can (and have already) improved, as well as where I question your conclusions. There are also points where I believe your post is technically correct if we think of security best practices in a vacuum but are cases where we have made reasoned choices to serve a greater client good that we believe is appropriate given the needs our partners and their clients have.
In the future, I know we can continue this conversation to the benefit of all our customers’ security - which is our ultimate goal.
Side note: Based on the information in your post and mining our logs, I believe (although don’t know with absolute certainty) that I have identified the device you were testing against. I’ve written the below as a general response but I’d be happy to discuss the particular device’s situation directly with you (provided, of course, that our partner agrees). My contact information appears at the end of this response.
Any penetration test must be evaluated considering the environment the device lives in. The IP phone on my desk at Datto is probably vulnerable to a DDOS attack (I haven’t checked, so please don’t consider this a challenge). However, my phone is designed to be used on an internal LAN, not on a public network. Therefore it doesn’t need the kind of security hardening other hosts do and in fact some best practices would render my phone unusable for practical purposes. At the same time, my phone does send calls over the WAN, so the pipe through which it does so (i.e., the encryption strategy used to secure my VOIP call) is subject to much higher security requirements.
Datto devices, much like my IP phone, usually live on private networks. With firewalls and strict port settings. Dattos only communicate outside that local network using strong encryption and authentication of the nodes in our cloud they are sending data to.
A potential criticism of your post is that it starts off assuming a number of calamitous events have already occurred and an adversary is on a privileged area of your network. If that’s the case, I’d say you’ve got bigger problems.
To be clear: I’m not hanging my hat on assuming the security of the LAN. I only raise this as a practical consideration our partners encounter every day. Not every client environment is the same. That said, Dattos can and must be secure even if the LAN is compromised.
Of course, this assumes the Datto is running current software…
Device OS Environment
The post says the Datto was running a 10.04 Ubuntu environment. Datto hasn’t shipped devices with 10.04 since early 2014. The OS version is past its end-of-life and has known flaws, including CVE-2012-0056, as your post correctly demonstrates.
100% agree this is a valid security concern in its current state.
Fortunately, there’s an upgrade path from 10.04 to 12.04. We now even offer a one-click upgrade right from the device itself. Screenshot below.
As a policy we never do serious updates requiring a device reboot without explicit permission from the partner managing the device.
Quite simply, updating this device’s software to the most recent Datto environment will prevent most of what was seen here. More specifically, the exploit used to gain root access is not present in 12.04.
Datto will proactively reach out to any partners with devices running 10.04 and strongly suggest that they run the above software update.
Datto devices ship with VNC enabled in the event partners require access to the device during installation. Much like VNC itself when first installed, devices ship with a default password (“Northern”) which I’m aware is well publicized.
Users should absolutely change the default VNC password upon installation of a Datto device. We provide this capability right in the user interface, and strongly encourage its use. Screenshot below.
That said, upon reflection and in light of your post, I’ve come to the conclusion that Datto should try to avoid situations where the result of inaction potentially creates a device vulnerability. Basically, we must assume that partners will forget to change the default VNC password for at least some of their devices.
Although the tools are already available to prevent this vulnerability, Datto’s standard operating assumptions need to be revised.
For devices currently deployed, we have updated all VNC passwords to be randomly generated strings (excluding devices where the partner has already affirmatively changed the default password). All new devices will have their passwords changed upon pairing with our cloud. The above is already complete for all devices in the fleet.
>However, VNC access isn’t very stealthy, and we wanted SSH access to the device instead.
By default, once a device has paired to our cloud, SSH is automatically disabled. The device must normally receive an “unlock” task from support to enable SSH. This is done for support ticket diagnostics or at partners’ request. Our policy is to issue a “lock” task to disable SSH on a device after a support incident has been resolved. However, in some cases, partners have requested devices be permanently unlocked for ease of access. I’ll address our solution to the later below.
>However, with a bit of research the default password to the account was found, “NorthernLight$”.
I’m not certain if this password was obtained by social engineering or otherwise. I’m now well aware this password is more widely distributed than I’d like.
If access was obtained by phone, I do know that Datto support usually deals with disasters where “time is of the essence” and, therefore, might seek to minimize phone bureaucracy for someone who clearly already has physical access to the device (e.g., the level of access that might be available by virtue of a security engagement).
However neither reality is justification for us allowing device access by an unverified caller nor allowing this password to be widely known. Datto messed this up and we’re working on fixing it.
Datto fell down here across the board. If the password was obtained from support, we shouldn’t have given any password out without 100% certainty that the caller was authorized to access the device. We also should not have allowed a standard SSH password to exist let alone become widely known, even though SSH was by default disabled.
I’ve asked our support manager to re-train every Datto support representative on proper phone verification procedures. That training will be complete by the end of next week. I’ve also updated our procedures so that permanent device unlock requests by partners will not be allowed. We are in the process of contacting partners who have unlocked devices in order to advise them of the need to disable SSH.
Finally, I’ve also taken the step of resetting SSH passwords for all devices in the fleet. This work is already completed. I did this because even though SSH is not enabled on devices by default, in this case SSH obviously was enabled and a more secure device-specific password would have provided another layer of protection.
Your post hits on something we spend a lot of time thinking about at Datto: how to serve the practical needs of our partners and their clients while maintaining the security of their data. Our encryption strategy is an example where I’d suggest we’re trying to strike that balance.
Agent level encryption, as you note, means that the encryption key must be available at some point in order to actually perform the encryption. Once a device is “unsealed”, we could certainly take one backup and discard the key. But that would force the user to re-enter their passphrase for every backup.
Given Datto deals in image-based backups, where it’s common to take incrementals every 15 minutes, forcing user input for every backup is a non-starter. The result of doing so would be that our clients take far less frequent backups. That outcome increases the risk to data in a different way than a security risk, but it’s a risk nonetheless. In our view, we have to take a balanced approach here.
On a technical point, I will note that what your post concludes is a master key is in fact not, it’s an encrypted master key. At device boot we generate a completely random key and salt, that is then used to encrypt the master key while in memory. Granted, the boot key is in memory as well as the salt and encrypted master key. But now an adversary must know all three. It’s not insurmountable if you’re already in the device shell as root, but it is another hurdle.
Although your post erroneously concluded that the master key had been obtained, we must consider again if we are making the appropriate tradeoffs in balancing security.
In light of your post, I’ve asked our security engineers to work with me to review how we approach key management and test that our trade-off decisions are indeed justifiable. If you have any input on how we can do better while still making a useful product, I’d welcome hearing them as part of this process.
Also, a patch has been created and is in review to change the behavior of snapctl “addAgentPassphrase” such that it requires an existing passphrase in order to add a new one. While this doesn’t stop someone from writing their own script to fetch the master key from memory and use it to create a new user key slot, it raises the barrier a bit. More on snapctl below.
Your post certainly highlights some bad things that can be done if an adversary gains shell access to a Datto (e.g., defeats access protections). Although the snapctl tool is critical to support in helping our partners diagnose difficult issues, I agree we need to restrict some of its capabilities. Particularly, I agree with your point regarding the system level command shell function in snapctl.
We should revisit the capabilities of the snapctl utility on Datto devices and verify each and every option’s benefit outweighs any potential risk.
I’ve asked support to assemble a list of the minimum snapctl issued commands required to handle partner support needs. In a device software update later this month, we’ll update snapctl to only accept that preapproved subset of commands. I’ll also work with our agent vendor to remove the underlying calls on the agent itself as soon as possible.
Minor point: The “snapctl screenshot” command may have been misunderstood. Although ominous sounding, it doesn’t take a screenshot of a live system. Instead, it boots the latest backed-up snapshot and takes a screenshot at the OS login screen only (as a visual method of backup verification). This has been a product feature in the SIRIS for some time, we call it “screenshot verification”.
Thank you again for taking the time to write such a detailed analysis of your work and conclusions. I hope that the above is an acceptable first step in what I hope to be an ongoing conversation with yourself and the community.
If you would like to discuss anything about this response, the device you tested, or even Datto in general, you’re welcome to contact me directly at firstname.lastname@example.org or by phone at (203) 529-4949 x250. That invitation applies to all readers of this response.
CTO, Datto, Inc.