November 6, 2017

Open letter to Datto partners regarding two reported product security vulnerabilities

Dear Datto Partners:

At Datto the security of your clients' data is always our top priority. We make every effort to avoid vulnerabilities in the first place, and when they do occur, we deal with them swiftly. This important responsibility is led by me along with my CISO and a team of security experts that monitor the evolving threat landscape and identify potential vulnerabilities proactively.

This document discusses two security vulnerabilities impacting only our agents that have been reported to Datto and which we are in the process of addressing for all customers. We have received no reports of any client device or Datto Cloud backup data being compromised as a result of these vulnerabilities. Additionally, for the vast majority of our partners, we expect you’ll find you already have sufficient standard network security practices in place to prevent these exploits from being used today. All that said, we are committed to providing as much information as possible so you can assess the current risk yourself.

Throughout this letter, I’ll discuss the way we make decisions about security generally and how we reevaluate prior decisions as the threat landscape changes over time.

Summary of Reported Vulnerabilities

The first vulnerability exploits the agent device pairing mechanism to allow a rogue user on the same local network as an agent, who is using a secondary Datto appliance or impersonating a Datto appliance, to pair with an agent.

The second vulnerability allows a rogue user on the same local network as an agent, who is paired using a secondary Datto appliance or impersonating a Datto appliance, to bypass agent command execution restrictions. The bypass allows non-whitelisted commands to be executed by the agent.

I’ll discuss each in more detail below.

Impacted Customers

At a high level, these vulnerabilities are only available to attackers that have already penetrated a business's network. Our partners are IT pros who understand that LAN access requires VPNs and segmented vlans. These threats assume those protections have failed.

In our line of business, even one vulnerable customer is an urgent problem. So let me be totally clear that we are working quickly to address the issues described below. However, as mentioned above, for the vast majority of our partners, we expect you’ll find your standard network practices will fully protect you from these vulnerabilities.

For those partners that do have vulnerable client environments, below are some LAN and network best practices we recommend. In combination, these would mitigate the vulnerabilities and eliminate any theoretical risk during the period between today and when we deploy a patch.

Datto’s Normal Security Vulnerability Evaluation Process

When we receive third-party reports of possible product vulnerabilities such as these, we always take those reports seriously and investigate every one. Additionally, we employ a team of internal pen testers to identify vulnerabilities proactively.

Once we conclude a patch is required, we work diligently to develop one. The time to create the patch depends on many factors including severity of the threat, prevalence, and the complexity of the required patch. If necessary, we have sufficient engineering resources to work around-the-clock on critical threats.

Once a patch is ready, we incorporate it into our immediate next software release. Software releases are generally available bi-weekly and include feature enhancements, bug fixes, as well as security patches. By default, devices receive all bi-weekly updates. Devices set to less aggressive update channels by their partners will still receive updates at least once per quarter. In the event a critical security threat was actually being exploited on the fleet, we would force an immediate update for all devices.

Timeline and Discussion of Current Disclosures

The two vulnerabilities referenced above were reported to Datto on October 25th by Continuum Managed Services. Since that date we have had a positive discussion with Continuum’s reporting engineers. Datto and StorageCraft engineers have also been hard at work designing and implementing patches. While we do not yet have a specific release date for the patch, it remains our engineering team's highest priority.

Continuum’s reporting engineers communicated early-on that they intended to submit a responsible security disclosure covering these vulnerabilities through the industry-wide standard security disclosure method called the Common Vulnerabilities and Exposure (CVE) process. Datto supports this. Through the CVE process, a report is typically filed but not made immediately public. During this time the company (here, Datto) has at least 45 days to remedy issues before public disclosure. This process protects users from undue risk. Datto’s view is the CVE process balances transparency with protecting users during the period between report, risk evaluation and patching.

As a general matter, Datto believes it makes users more vulnerable when the details of an as-yet-unpatched vulnerability are disclosed publicly, particularly if the exploit is unlikely to be discovered in the interim and the company is working diligently to address it. In the case of these particular vulnerabilities we feel strongly that the customary CVE process is entirely appropriate.

Continuum, however, has notified us that they intend to independently publish a "customer security update" on this issue prior to our patch being made available. We support their right to do so. Still, we are concerned that their update may focus only on worst-case scenarios, not take into account that the vast majority of our partners are IT experts with standard network security practices in place to prevent these exploits from ever being used today, and could only offer limited mitigation advice.

As such, this letter discloses some details of the vulnerabilities and provides clear mitigation steps if you determine you are at risk. While we recognize that any early disclosure may enable some attackers, based on the circumstances we feel it is necessary to now balance that risk with your need to evaluate your own exposure.

Issue #1: Agent Pairing

The first vulnerability relates to how agents establish trust with a device and how a potential adversary using a second Datto device (or impersonating one) can successfully pair with an agent.

Threat landscape

Datto's design for agent-to-Datto-device communication assumes a secured LAN. We have made this reasonable assumption because the requirement of a secured LAN for backups has existed for nearly every backup solution, pre-dating Datto by many years. Traditional backup solutions send data via unencrypted iSCSI, to network shares, or other backup targets, so without proper security an adversary can observe any block backups moving around your network. If the backup LAN is not well secured, the network is at risk for even more disruption than just the sniffing of backup data. Therefore, Datto expects backup devices to participate in a secured network environment.

When we designed our solution several years ago, another requirement was easy upgrades: as a client's data grew it had to be easy to switch to a larger or newer backup device. So we are “permissive” in allowing a new Datto device to replace an old one, and allowing multiple devices on a LAN to actively backup protected machines on that LAN. An agent pairs and communicates with any other backup LAN device appearing to be a valid Datto. The pairing process requires specific information present only within our proprietary software or obtained directly from Datto Cloud. At the time of design, this was considered safe given the assumption of a secure LAN in the first place.

Today’s operating environment

However, now there are thousands of Dattos and hundreds of thousands of agents in the world, Datto is a bigger target for adversaries, and the overall security risk profile for SMBs has grown. Therefore, we need to improve how our agents and devices verify each other’s identity, even within a trusted network, to prevent imposter devices or device impersonation as demonstrated in this exploit.

Threat description

If we assume a Datto device, or a host that can impersonate a Datto device, has been compromised within the backup LAN, then Datto agents are currently vulnerable to rogue pairing: the ability for an attacker to pose as a new Datto device and request data, usually in the form of a backup, be sent to it.

Rogue pairing, in practice requires:

  • The rogue second device or an adversary impersonating a second device to already be on a trusted private network.
  • For an adversary seeking to obtain an agent machine’s data, proper use of the API commands that the agent provides to take a backup.
  • Enough time to copy a full backup (which can be hours).
  • That the legitimate Datto device does not attempt to re-pair with the agent during the above time.

Threat response

We are working with both StorageCraft and on our own agent to release an update that vastly improves our device-agent pairing and verification process. We expect to release a Datto Windows Agent (DWA) update that addresses this problem within the next 30-days, hopefully much sooner. We are partnering with StorageCraft to address the equivalent vulnerability in ShadowSnap as quickly as possible.

We also recommend our partners read and adhere to our recommendations regarding secure device deployment.

Examples of secure deployment practices that already prevent the vulnerability described above from being exploited (and provide broad protection against unknown attacks):

  • Prevent non-Datto devices from communicating with agent machines over port 25566 or port 25568.
    • Most of our partners utilize Windows domains. In most cases we see, all protected machines (e.g., machines running our agent) are joined to the domain and inherit its group security policy. Setting a domain network rule to only allow protected machines to receive communication over port 25566 or 25568 originating from the legitimate Datto device’s local IP addresses prevents a second device or an attacker posing as a device from talking to the agent’s API.
    • Every Windows operating system has Windows Firewall built in. Partners can individually protect agents by setting an equivalent local firewall rule that evaluates prior to an implicit deny, as demonstrated below.
      --------------------------------------------
      Execute below on computer running agent...
      --------------------------------------------
      C:\Windows\system32> netsh.exe advfirewall firewall add rule name="Only Datto
      device’s IP should communicate with agent ports" protocol=tcp dir=in
      enable=yes action=allow profile=Any localport=25566,25568
      remoteip=<span style="background-color:yellow;font-size:14px;font-family:monospace">{INSERT_DATTO_DEVICE_IP}</span>
    • Most routers / UTMs our partners deploy can similarly enforce this rule. The capability is often referred to as ACLs.
    • Some advanced layer-2 switches support greater LAN access controls. This may allow you to deploy vlan or switch port ACLs to permit this traffic only from your datto device.
  • Prevent unknown hosts from joining the backup LAN your important workloads are on.
    • Modern routers and switches support port based security allowing for host verification (through legacy MAC address filtering, modern 802.1x radius authentication or otherwise) before hosts are added to a privileged vlan.
    • Default rules should not allow unknown hosts to be put into the same vlan as protected hosts.
    • For remote VPN connectivity, add two factor authentication if connected clients have access to the backup LAN.

Issue #2: Agent Command Execution Whitelist Circumvention

Background

The only sure-fire way for a business to secure its computers is to never turn them on. But once they are on and connected to a network, BDRs and RMMs are important tools MSPs use to keep them protected.

Each of these important tools requires an agent on the protected machine. Even though any software carries a theoretical risk, agents are particularly attractive attack vectors because they interact with important data and, by design, usually require elevated privileges. But those of us familiar with these tools know the benefits they bring outweigh the risks, so long as the risks are properly managed.

Datto’s agents work deep inside the Windows kernel and we know you entrust them to do important work on your behalf. We built our Datto Windows Agent (DWA) to backup any Windows operating system shipped in the last 20 years and future versions seamlessly. Our partner StorageCraft built their agent to meet the same challenge.

Why do agents have command execution?

An agent can only control or know so much about its environment in advance. Software like an agent or RMM - designed to be deployed to remote locations, into an infinite number of different OS and application environments, and support every edge use case - is going to be ultimately more reliable and supportable if it can interact with other software in the environment when appropriate and necessary. That’s why both ShadowSnap and DWA (and all RMMs we’re aware of) provide for limited command execution.

Balancing functionality with risk

While we stand behind the need for Datto devices to have limited command execution authority over their agents, we naturally want to limit the damage a compromised device or agent could inflict. Basically, our goal is to reduce the agent surface area a successful adversary could exploit while also allowing enough flexibility for the agent to be reliable and supportable in thousands of different environments.

That is a tough balance to strike, particularly for image-based backup software.

Over time we’ve aligned around a command whitelist approach that we think balances risk with functionality. The whitelist design requires that command execution requests sent by a device to an agent will only be executed by the agent if they are whitelisted in advance. It also specifies, as an additional precaution, that devices themselves should not accept requests to, nor attempt to, issue non-whitelisted commands to their paired agents.

Below is the status of command execution whitelist enforcement on Datto’s device and paired agent software:

  • Datto devices
    Since early 2016 Datto device software has limited agent command dispatch for shell users, including Datto’s own support team, to only whitelisted commands.

  • Datto Windows Agent
    Datto’s Windows Agent was designed with the ability to enforce a command whitelist. Recently a DWA vulnerability was identified where a malformed primary whitelisted command input could allow a secondary non-whitelisted command to be executed. The Datto Windows Agent team has already addressed this bug in the latest DWA software and if you’re running a version of DWA greater than 1.0.4.0, your DWA agent will auto-update. For the limited number of partners running a pre-1.0.4.0 version of DWA (which lacks auto-update), you should immediately update to the latest version posted on downloads.datto.com.

  • ShadowSnap agent
    Datto has been and continues to work with Storagecraft to update their software to implement command whitelisting. Their engineering team is rigorously engaged in completing that work.

To summarize: Continuum engineers have identified two security vulnerabilities and reported them to Datto. Datto has evaluated the vulnerabilities using our standard process and agree a patch is appropriate. We are actively working on this patch. We know of no actual exploitation of these vulnerabilities, and we expect most of you already have appropriate network safeguards in place to avoid exposure. However, we do know the vulnerabilities are imminently being made public. Therefore, we are providing this disclosure so that you have as much information as possible and can take any additional steps to protect client data given the vulnerabilities will soon be public.

This incident has provided me with difficult "judgment calls" forcing me to balance our commitment to transparency with the best ways to protect our partner community. We pride ourselves in putting our partner's interests first, so I've written this always considering the information and detail that you, our partners, would want us to provide. I hope I've achieved that here.

Please feel free to reach out to me directly at rgibbons@datto.com or by phone at (203) 529-4949 x250 with any questions, comments or concerns.

Sincerely,

Robert Gibbons
CTO, Datto, Inc.