Datto’s Response to Log4Shell

On Friday December 10, 2021 news of active exploitation of a previously unknown zero day vulnerability (CVE-2021-44228) in a common component of java-based software, referred to as log4j, became widely known. The extent to which this software package is integrated into the world's technologies and platforms is still being discovered, making response a fluid activity for any security program.

Situation Report

Datto has not assessed any material exposure to the log4j vulnerability that would impact the safe use of Datto products at this time. Should this assessment change, we will update Datto partners immediately.

Datto has leveraged the intervening hours since the public disclosure of the exploitation to mount a comprehensive assessment and response. This post will detail for you what Datto has done to assess and respond to this vulnerability, what we continue to do to assure the safe use of our technology, and offer some information and intelligence from our own security program that you can use to help protect yourselves and your SMB clients.

The Vulnerability

For those who remember, this log4j vulnerability will invoke memories of ShellShock in 2014. Drawing on that analogy, you can conclude that this log4j vulnerability is potentially the most impactful critical vulnerability that we have seen this year.

Exploitation of the vulnerability requires a single HTTP request containing malicious input from anywhere in the world, to an internet-facing server that is running a vulnerable instance of log4j. The result is a full system compromise and the exploit requires no authentication. This is as bad as it comes, especially given how widely this common software component is deployed.

At this time, others in the security community have done fantastic work writing up how the vulnerability works. Rather than rehash the details, I would have you review the comprehensive write-ups released by CloudFlare and SANS Internet Storm Center.

Vulnerability Response

Datto’s Engineering teams have not identified any material exposures to the vulnerability, and are confident in the safe use of Datto products. While we consider our initial response complete, we remain in a state of active monitoring and readiness to respond.

This situation is evolving and we fully expect news of additional affected technologies to become known over the coming days and weeks ahead. All technology professionals will need to monitor for the latest developments and continually reassess their exposures.

Our top priority was to complete an initial comprehensive assessment and response. This has been completed. The focus of those activities centered around the following:

• Assessing usage within Datto products
• Inspecting infrastructure systems in our asset inventories
• Researching vulnerable third party technologies
• Inventorying Datto’s third party vendors to engage them and understand their response

Response Timeline

When analyzing security events, it is helpful to review the response effort as a timeline. Key moments from Datto’s vulnerability response timeline are provided to illustrate how our response progressed and to give a sense of its depth and scope.

By 10:00 AM ET on Friday

• The Datto Information Security team started initial triage and after a short period of time, determined this was a potentially serious issue requiring foral response.

By 11:00 AM ET on Friday

• Datto CISO declared a security incident and leveraged pre-established processes to redirect Engineering & IT resources onto the active response effort
• Datto’s Third Party Risk team began assessing vendors for potential vulnerability
• Software Security Champions are gathered and began assessing Datto Products for potential impact under the guidance of Datto’s Application Security team

By 12:00 PM ET on Friday

• Datto’s Security Operations team
  • confirmed that in place logging is sufficient to observe malicious requests
  • started creating new alerting rules to triage scan events in real time
  • confirmed all scan activity was not successful; did not result in C2 domain resolution or outbound connections

By 2:00 PM ET on Friday

• Datto’s Offensive Security team had a working scanner running against all Datto internet-facing web presence automatically assessing for vulnerability

By 3:00 PM ET on Friday

• Datto’s Offensive Security teams scan was completed and has found no vulnerable instances across all of Datto’s web-facing infrastructure (all products and clouds)
• Datto’s Third Party Risk team leveraged open source information and our continuous vendor monitoring platform to identify vendors that are potentially susceptible and started contacting them

By 4:00 PM ET on Friday

• Initial assessment is nearing completion.
• Despite a few Datto products having log4j in use, it was determined that those instances were
  • not running a vulnerable version;
  • not internet accessible and at immediate risk;
  • protected by compensating controls;
  • already upgraded; or
  • were running with the configuration workaround applied

By 12:00 PM ET on Saturday

• The broad response team regrouped and started systematically re-verifying our assessment findings from Friday, and integrating new information that came out overnight, to help us further respond.
• No additional instances were identified, and continued exploitation attempts were not successful, leading Datto to persist in our prior day’s conclusion that Datto products remain safe for use at this time.

Continued Response

Datto Engineering and Information Security teams remain in a state of active response and readiness in order to:

• Triple check our initial assessments
• Monitor for additional intelligence emerging about the scope of technologies impacted
• Reassess for exposure as necessary
• Leverage commercial and open source threat intelligence to monitor for changes in attack patterns and adapt response activities if/as necessary
• Continue to engage with vendors to assure our Third Party (our partners’ Fourth Party) risk is understood and appropriately managed

Threat Intelligence

Datto Security Operations team began actively monitoring the environment for malicious payloads.

Detection Opportunities

Indicators of enumeration and exploit attempts can be found in logs of software that leverages log4j.

Datto’s partner in intrusion monitoring, Red Canary, released guidance that the following exploit trigger strings are expected in various portions of HTTP requests (e.g. user-agent, Username, URI, etc.):

• ${jndi:ldap://
• ${jndi:ldaps://
• ${jndi:rmi://

Datto's Intrusion Monitoring team has observed payloads inserting the ${jndi: exploit strings in many different portions of web requests.

Evasion

Security researchers caution that monitoring which relies solely on ${jndi: pattern matching is incomplete as there are a number of evasions. One such example from Brandon Forbes, and retweeted by Katie Nichols of Red Canary, can be viewed here.

Sources

It’s now widely reported that TOR exit nodes are generating a lionshare of the scanning activity and Datto’s monitoring confirms this. Consider blocking all inbound connections from TOR exit nodes.

Command and Control (C2)

Below you will find a list of the C2 servers that Datto has observed.

• dnslog[.]cn
• interactsh[.]com
• interact[.]sh
• burpcollaborator[.]com
• eg0[.]ru
• bingsearchlib[.]com

It is possible that these are being used for legitimate research purposes, and also for malicious purposes. At this time, we suggest that you block inbound requests with these C2’s payloads and outbound requests containing these C2’s.

Payloads of Interest

Using our logs and open source intelligence (OSINT) we identified multiple C2 IP addresses serving malicious shell scripts. These are the first of what will be many, but studying them can be informative about what is to come. We feel these should be paid special attention.

lh.sh and lh2.sh

The shell scripts are not being served as of this writing and that makes offline analysis difficult. Based on OSINT we believe that these files are associated with active attempts to:

1. Recruit systems for participation in a DDoS botnet
2. Recruit a system for participation in Coin Miners

Looking Forward

The days and weeks ahead will be challenging as exploits mature and evolve, more becomes known about common technologies that have this vulnerability embedded within them, and more third party disclosures come out regarding technology susceptibility.

Coin miners and DDoS botnets are just the start of what we will see in the days ahead. It is reasonable to predict that mass exploitation will occur in the near future. It is also likely that ransomware affiliate groups will include this exploit in their playbooks and ransomware threat actors will embed exploitation of this vulnerability into their malware kits.

The good news is that there is still time to take action. Leverage the information contained within this blog to help protect yourselves and your SMB clients, and be assured that Datto is vigilant.