Log4J Exploit Detection (CVE-2021-44228)

If you are reading this then I assume you have already heard about CVE-2021-44228, the Remote Code Execution (RCE) vulnerability affecting Apache Log4j, the Java logging library much of the internet uses on their web servers. While many blogs and comments have posted methods to determine if your web servers/websites are vulnerable, there is limited info on how to easily detect if your web server has indeed been exploited and infected. But first, a quick synopsis:

CVE-2021-44228 Apache Log4J RCE

• First, as most twitter and security experts are saying: this vulnerability is bad. Real bad. Many prominent websites run this logger.

• RCE = Remote Code Execution. The attacker can run whatever code (e.g. malware) they want on your webserver by sending a web request to your website with nothing more than a “magic” string + a link to the code they want to run.

• Affects Apache web server using vulnerable versions of the log4j logger (the most popular java logging module for websites running java).

• Vulnerable versions: 2.0 - 2.14.1 (Recommend updating to 2.17.0)

Has anyone tried to exploit my web server?

Typical behaviors to expect if your server is exploited by an attacker is the installation of a new webshell (website malware that gives admin access to the server via a hidden administrator interface). Apache would run curl or wget commands to pull down the webshell or other malware they wanted to install.

Luckily, there are a couple ways to detect exploit attempts while monitoring the server to uncover previous exploit attempts:

1. Review apache logs for jndi:ldap, jndi:rmi or jndi:dns. These are the magic strings that cause the logger to go haywire and follow/execute the url that follows it.
2. Scan /var/log with yara signatures matching some of these indicators.
3. Scan the webserver for generic webshells.
4. If you have EDR on the web server, monitor for suspicious curl, wget, or related commands. Likely the code they try to run first following exploitation has the system reaching out to the command and control server using built-in utilities like this.

NOTE: If the server is exploited by automated scanners (good guys are running these), it’s possible you could get an indicator of exploitation without follow-on malware or webshells. Some research scanners exploit the vulnerability and have the system send out a single ping or dns request to inform the researcher of who was vulnerable.

Datto Partners: RMM Component

Datto has released both a Datto RMM component for its partners, and a community script for all MSPs that will help you use the power and reach of your RMM, regardless of vendor, to enumerate systems that are both potentially vulnerable and that have been potentially attacked. The tool can also attempt to protect against subsequent attacks by applying a known workaround.

Please don’t hesitate to reach out to our team of experts for help.

UPDATE 12/22:

Over the last week we have seen a lot of scanning activity from security scanners, wide-scale exploit activity from Russian and Ukrainian IP space, and many exploits of systems ranging from Elastic servers to custom web services.

We have updated our log4shells scanner to include better coverage of obfuscation methods and also depreciated the now defunct mitigation options that apache previously recommended. The enviroment variable LOG4J_FORMAT_MSG_NO_LOOKUPS or log4j2.formatMsgNoLookups=True cli argument will not stop many attack vectors.

In addition, we expanded the scanner to look at all drives (not just system drives or where log4j is installed) and recommend running it again if you haven’t recently.

1. Finds any .jar files with the problematic JndiLookup.class
2. Scans the system for compressed and uncompressed .log files with exploit indicators related to the log4shells exploit.
— Primary path on Linux and MacOS is: /var/log
— Primary paths on windows include $env:SystemDrive\logs\, $env:SystemDrive\inetpub\, as well as any folders that include the term java, log4j, or apache.
3. Utilizes open sourced yara signatures against the log files as well.

Due to how many implementations there are of log4j embedded in various products, it’s not always trivial to find the version of the log4j extension. The easiest way is to look at the file or folder name of the .jar file found with the JndiLookup.class but this isn’t always present. Some products require specific vendor instructions.

UPDATE 12/14:

Attacks continue to be thrown against vulnerable apache servers, but this time with more and more obfuscation. Attackers appear to be reviewing published intel recommendations and testing their attacks against them. We’ve updated our log4shells/log4j exploit detection extension significantly to maneuver ahead.

Regex matching in logs can be tough to get right when actors obfuscate but it’s still one of the more efficient host-based methods of finding exploit activity like this. The above shows various obfuscations we’ve seen and our matching logic covers it all. Our approach with rules like this is to have a highly tuned and specific rule with low false positives and another more generic rule that strives to minimize false negatives at the cost of false positives. Our hunters generally handle triaging the generic results on behalf of our customers.

In addition, generic behavioral monitoring continues to be a primary capability requiring no updates. If apache starts running new curl or wget commands (standard 2nd stage activity), it will be reviewed.

We’ll keep monitoring as the situation evolves and we recommend adding the log4j extension to your scheduled scans.

UPDATE 12/12:

Worked with a couple of our partners late last night and updated our extension for windows-based apache servers as well:

One issue with scanning logs on Windows Apache servers is the logs folder is not standard. Our extension will therefore look in <em>[DriveLetter]</em>:\logs\ (aka C:\logs\) first as it is a common folder but if apache/httpd are running and it’s not there, it will search the rest of the disk.

Determining if there are .jar files that import the vulnerable code is also conducted. Suggestions from partners in the field looking to query for an environment variable called log4j2.formatMsgNoLookups can also help but understand there are a lot of implementations where this value could be hard coded and not in an environment variable.

IMPORTANT: A lot of activity we’ve seen is from automated scanners (whether researchers or otherwise) that do not follow up with webshell/malware delivery or impacts. The ease of exploitation of this bug can make this a very noisy process so we urge everyone looking for exploitation to look for other indicators of compromise before declaring an incident from a positive match in the logs. Please contact us if you’re having trouble on this step.

Not a Datto partner yet? Reach out to request a demo today.