Thu, Sep 12th, 2019

MSP Intrusion: Local Network Backup Solutions Targeted for Ransom Success

OPEN RESPONSE AND NOTICE TO MSPs:

Ten days ago, an MSP made the community aware of an intrusion where an attacker targeted a Datto backup appliance deployed on an end user’s local network in order to increase the likelihood of successfully extorting ransom payment. The affected MSP’s community posts have created broader awareness that hackers continue to attack backups as part of ransom attacks. There is now a deep desire for more information amongst MSPs that were unaware of this fact. In this case, an attacker resident on the end-user local networks with MSP admin credentials, opted to delete all the backup points on local appliances as well as offsite in the cloud. The paradigm of MSPs administering global deletion from local appliances in secure non-public networks has operated safely for many years without issue. However, there comes a time when such paradigms need to be revisited given new information. MSP and end user networks are being accessed for malicious use at increasing frequency. Datto is revisiting this paradigm and adding safeguards given this shift in the threat landscape.

This letter provides insight into what occurred in this instance and what Datto has done and will continue to do to help MSPs avoid these negative outcomes. We believe a transparent approach leads to visibility and accountability that we want to foster within the MSP ecosystem.

Situational Details:

At this point, we have a set of conclusive facts that are known and safe to share. Ultimately, this threat actor did not access the MSP end user networks through any Datto technology. As our investigation led the MSP in another direction, we have handed this incident back to the MSP and alerted another vendor who will assist the MSP taking over the investigation, perhaps with continuing transparency following a thorough assessment. The basic details of the attack that follow are intended for community education and are in no way an assignment of blame.

An MSP had five of its customer environments experience a compromise on August 24 and August 25. The ransom attack experienced by those end customers did not similarly impact the MSP’s network. The MSP had many customers that were not affected by the ransom attack. Intelligence and facts gathered are as follows:

  • The attacker used a ransomware variant known as GlobeImposter 2.0, for which there is currently no available decryptor.
  • The MSP had a technician operating without 2FA on at least two of their channel vendor technology accounts.
  • Several of the MSP’s backup appliances had a shared admin credential configured across them. The local appliance web interface was enabled for these appliances.
  • It is unknown to Datto at this time how the attacker gained access to the MSP technician’s account password.
  • The Datto Partner Portal alerted the MSP to a failed technician account login attempt from IP 212.92.122.66 on August 24th. The login failed due to an incorrect password.
  • Datto was made aware by the MSP of indicators that the information stored in the MSP’s documentation and password management facility was accessed by the attacker to:
    • find a point of entry into and access the five customer environments
    • discover the shared login credential for local BCDR appliances
  • Through information sharing efforts, it became known that there were malicious login attempts, seen across channel technology providers, using the MSP technician’s account.
  • There is evidence that the MSP’s documentation and password management facilities were accessed by an attacker using the technicians account on August 20th from the below set of IPs:
    • 212.92.122.66
    • 89.238.154.163
    • 82.102.27.114
    • 196.52.34.16
    • 212.92.123.45
    • 80.127.116.96
    • 176.10.99.200
    • 37.139.8.104
    • 51.15.49.134
    • 46.165.245.154
    • 5.2.74.80
    • 185.220.101.49

Datto Action:

The role of Datto BCDR appliances is that they are the authority for what is to be stored and retained local to the appliance and in the Datto Cloud. This appliance-based state tracking paradigm puts the MSP in complete control of backups and restore points from a central location. An MSP can delete unneeded backup points from the local appliance to free up storage space, stay within the offsite limits of their service plans, decommission a legacy system, and many other valid workflows. This feature gives maximum flexibility to the MSP and has historically been safe to reside on the appliance as it exists within a local area network that has layers of security and access controls surrounding it and protecting it from attack.

Nearly a year ago, Datto took steps to help MSPs protect the vast majority of offsite backups from malicious deletions when MSP or end user networks are compromised. We instituted a delay in replicating deletes between our primary and secondary datacenters. While this has aided a number of Datto partners to date in recovering from targeted attacks and accidental deletions, this protection only helps when the appliance has secondary replication enabled. It does not extend protection to entry level non-SIRIS BCDR appliances that do not offer service plans with secondary replication. This is why two of the five devices impacted in this incident did not have recoverable data offsite.

To raise awareness and educate on this point, we have been requesting MSPs follow security best practices. We know from experience that those compensating measures do help MSPs avoid these bad outcomes. Read more here: https://www.datto.com/best-practices-for-a-secure-bcdr.

In parallel to education, we opted not accept this gap in our fallback layer of protection and took steps to help MSPs protect themselves when devices are non-compliant with best practices. The below list of enhancements and processes have already been delivered as part of those ongoing efforts:

  • Added a BCDR feature to allow MSPs to disable the local appliance WebUI, thus restricting appliance access to only the Datto Partner Portal. If you have 2FA enabled on your relevant Partner Portal accounts then your appliance effectively has 2FA for logins as well
  • Assessed the salted and hashed versions of user passwords of all Datto Cloud products (not appliances) for presence of known compromised or easily guessable/crackable passwords. We forced a reset of thousands of passwords as a result of this action protecting MSPs from common attacker techniques.
  • Began notifying partners in real-time via email of Datto Partner Portal login attempts from new IP addresses. This has helped multiple partners detect anomalous activity indicating they are being targeted
  • Began monitoring authentication attempts for simple signs of maliciousness and have been able to alert multiple MSPs to signs of targeted attack before such an attack can be mounted against them
  • For Datto RMM, we have disallowed logins from previously unseen IP addresses until the MSP validates the login via an email based workflow
  • Began encouraging increased adoption of 2FA in the Datto Partner Portal. We now have over 70% of active users having 2FA enabled
  • Began an outreach program to all BCDR and RMM partners that have at least one user on their account that does not have 2FA enabled
  • Introduced a feature called Trusted Networks. MSPs that are concerned with usability implications of 2FA on productivity can now define a set of trusted IPs that prompt for 2FA less frequently. MSPs can also leave all IPs as untrusted and it will require 2FA on each login from anywhere. Please consider how you want to configure this feature in a threat model where attacker is already resident in your network.
  • Re-introduced a feature that had been temporarily disabled to support material improvements of the authentication platform on Tuesday, September 10, 2019 that lets MSPs view the 2FA status of all their users from the Manage Employees page in Datto Partner Portal

But There Needs to Be More:

We laid the groundwork, but there is more we can do to aid MSPs in protecting themselves. There are currently multiple engineering teams at Datto dedicated solely on designing and implementing added layers of defense that will make Datto MSPs more immune to malicious backup deletions.

Examples of these planned future enhancements include, but are not limited to:

  • A set of BCDR and Datto Cloud enhancements, such as the ability to:
    • retain a copy of agent deletions for any appliance for a period of time in the Datto Cloud to facilitate full and faster restore (this is a top priority with active effort)
    • only allow agent deletions for appliance logins originating from the Datto Partner Portal
  • A set of authentication security enhancements, such as the ability to:
    • extend support for alternate TOTP solutions in the Datto Partner Portal to drive 2FA adoption toward 100%
    • block all untrusted IP and require 2FA for all trusted IPs on each login
    • allow bring-your-own 2FA via SAML/SSO integrations
    • automatically disable all user accounts that are inactive for an extended period of time

Datto will not rest until MSPs are made safer from this endemic threat. We've been working diligently on these enhancements in the background. While we want nothing more than to deliver this functionality quickly, we also need to take the time to get it right.

In the current threat landscape, it is not productive for MSPs or MSP vendors to be divided. Our entire community shares a common set of threats and our mutual success depends on sharing intelligence. Our hope is that this transparency will better inform the actions we can take together to combat these threats and give you confidence we are continuously working to help you combat these risks and not accepting the current state. Please reach out to me anytime at rweeks@datto.com.

Stay safe.
Ryan Weeks
Chief Information Security Officer