Network defenders are racing to counter multiple threat actors determined to exploit the Log4j vulnerability (CVE-2021-44228) known as Log4Shell. And longer term challenges are likely to arise due to the breadth of Java software, and the associated Log4j logging library, being deeply embedded across the enterprise. As the most popular development language for enterprise applications, Java exists in the components powering numerous web apps and unseen backend systems. Here we highlight short- and long-term steps organizations can take to protect themselves. While these steps will solve issues, the Log4j vulnerability shows the importance of moving to persistent hunt operations.

There is no single solution to all the challenges posed by this vulnerability. Several Log4j patches have been issued (including versions 2.15, 2.16, and 2.17) because initial patches introduced further problems. The Log4j vulnerability is a reminder that simply blocking indicators of compromise (IOC) is insufficient for effective security operations. Moreover, when enterprise software libraries are impacted, and organizations have been forced into a cost versus benefit approach to security information and event management (SIEM) licensing, the SIEM cannot be the end-all-be-all to the organization’s security operations. 

Part of what’s needed to address such vulnerabilities in the future is a new effort to improve software security and supply chain risk management with a software bill of materials (SBOM). An SBOM is a list of software components in a digital product. Universally mandated SBOMs would empower all players in the digital ecosystem to reduce risk and increase security and competition. While many improvements are still needed to increase product security, a universal SBOM mandate is one step with the power to incentivize industry to develop secure software.

Universally mandated SBOMs would also provide consumers the transparency they need to comparison shop on the vulnerability of software components. Currently, consumers cannot determine what software is included with the products they buy. As a list of all components used to make a software product, an SBOM would reveal which open source or proprietary code objects are embedded in the software. Knowing this would allow end users to better understand how bad actors might target new software by attacking old code elements that harbor unpatched vulnerabilities.

Much data has been shared since the Log4j vulnerability was made public and there’s no doubt organizations are overwhelmed trying to search for systems, patch vulnerable servers, block malicious activity, and ensure they conduct a full assessment, all while also supporting day-to-day activities. A graphic by the Swiss Government CERT “The Log4j JDNI Attack and How to Prevent It” can help organizations better understand the attack. 

Here are steps that organizations should have already completed or should plan to complete very soon:

1)    Implement Sensor Blocks – Web Application Firewall (WAF), Intrusion Detection System (IDS), and firewall blocks should have been implemented on all perimeter and internal devices. While your options may vary by vendor, a running GitHub Gist has been created to provide vendor-specific guidance and links to the most updated KBs.

2)    Disable Log4j – In cases where Log4J logging can be disabled, placing a property string within the src folder will disable the Java Naming and Directory Interface (JNDI). Under src/main/resources, create or edit spring.properties and add spring.jndi.ignore=true to the file. This will disable Log4j from running.

3)    Identify and Patch Vulnerable Versions – An update for the log4j vulnerability has been posted. When possible, an update should be applied from Apache’s website or GitHub (Before applying, don’t forget to verify PGP and SHA512 signatures). It’s incredibly important that security teams ensure version 2.17 is the patch applied. Version 2.16 completely removed the JNDI library. Version 2.15 is vulnerable to the same exploit as 2.14 and CVE-2021-44228.

4)    Disable JNDI Lookups – In the case where applications need JNDI to run, organizations should consider disabling JNDI lookups from hitting the LDAP servers. LDAP serves an important aspect within enterprises of providing direction to data—however, in this case, attackers can exploit JDNI lookups to further enumerate networks, collect data, and discover traversal paths.

To disable JNDI lookups, first disable all Java Services on your respective system, then, Disable Logging in System Properties for both the client and server components for versions >=2.10 by either

a.     Add -Dlog4j2.formatMsgNoLookups=true to the startup scripts of Java programs.

b.     Set the following environment variable: LOG4J_FORMAT_MSG_NO_LOOKUPS=”true”.

c.     Alternatively, disabling the JndiLookup class from the classpath will also work. To do so, a command such as zip -q -d log4j-core-*.jar org/apache/logging/log4j/core/lookup/JndiLookup.class will remove the class from the log4j-core.

5)    Disable Remote Codebases – Log4Shell is possible due to how the JNDI library uses LDAP to load remote Java classes. Disabling the remote calls to Java libraries is ideal—however, as always, this should be tested and observed closely, as this may break some legacy Java applications.

6)    Perform Scan with Updated Vulnerability Management Templates – Ensure that vulnerability scanners have credentialed access to scan as some may not pick up the vulnerability without it. Some organizations may be able to use a few open-source tools such as Grype and Syft. Both are trusted tools from Anchore but may require the installation of Go. Scan for evidence.

7)    Perform Searches and Analysis of All Security Logs for Evidence of Enumeration or Compromise – Leverage other security tools and look beyond the SIEM if logs are not being aggregated or consolidated into an enterprise log management solution. A conscious effort should be made to consolidate logs not going to SIEM into a central location if possible.

Special attention should be paid to

${jndi:ldap:/}
${jndi:ldaps:/}
${jndi:rmi:/}
${jndi:dns:/}
${jndi:iiop:/}

Emerging Threats maintains a list of Suricata rules, including 12 rules that alert on LOG4Shell.

Many attempts may involve obfuscated code and are not easily searchable by simply searching files. Tools such as Florian Roth’s log4j scanner consider the obfuscation attempts. Organizations should consider writing/moving logs to a central location and cron-ing the Python script. An example may be ‘5 * * * *  -a source/file/path new/file/path’ should help keep files in sync then ‘ 5 * * * * /path/to/python3 log4shell-detector.py —defaultpaths >> .log4shelloutput.log’.

8)    Consolidate, Communicate, and Disseminate Updated Threat Intel Associated with Log4j – With the amount of info and intel that is being released by numerous sources it is essential that stakeholders and system owners are receiving the latest intel and applying the updated patches applicable to their applications and systems. An organization should designate a person or team to sift through all the potential disparate and extract the intel that is most relevant to their respective environment, including applications and systems.

9)    Track All Remediation and Mitigation Efforts and Tasks – Related to the amount of intel being released are the associated tasks that need to be performed to remediate the vulnerability and enable additional protection mechanisms to increase the security resiliency. Organizations should be tracking in a master Log4j designated ticket or case management solution what applications and systems are being worked on, and by what teams, and any, and all issues that are preventing tasks from being completed. This helps ensure that there are no gaps in remediation and mitigation coverage that an attacker could leverage to compromise.

10)   Continue to Apply Up-To-Date Blocking Measures – Ensure systems are receiving the most up-to-date IOCs. While IP blocking is a game of whack-a-mole a few sites like Zeek and GreyNoise are providing up-to-date info and helping rule out malicious attempts from benign internet scanning.

11)   Monitor LDAP Traffic – Once all immediate and short-term mitigations have been applied, organizations should monitor for malicious LDAP traffic against known/allow list traffic.

12)   Move Vulnerable Systems Behind Additional Firewalls – Finally, if systems cannot be updated at this time, all vulnerable systems should be moved behind additional protection.

While the list above may be overwhelming, we provide it as a template and not as a checklist. Organizations will need to choose the level of risk they are willing to accept that matches their objectives. New techniques to exploit this vulnerability, using techniques like command obfuscation, are being observed every day. It will be very difficult to maintain detection rules quickly enough to fully protect vulnerable assets. Also, the fact that a patch contains vulnerabilities that are being actively exploited, and there is a chance that future patches may as well, means that patching alone is not a silver bullet for protection. For these reasons, organizations need to take the position of “assumed breach” of vulnerable assets.

Upon initial exploit, threat actors will have achieved a foothold using a myriad of available tools and can conduct the typical activities of reconnaissance, privilege escalation, lateral movement, and persistence. However, not all threat actors will do this immediately following initial exploit. It is likely that patient actors will lie in wait and avoid detection. This is why persistent hunt is necessary to detect delayed threat actor activities within the network and on previously exploited hosts.

Threat hunters should consider all previously vulnerable assets as high-risk and conduct persistent, frequent, and thorough hunt activities on all these assets. The hunt should include all networks, assets, and resources that are adjacent to the high-risk assets in an attempt to identify lateral movement, reconnaissance, persistence, or exfiltration.

Much of the activity that is being observed is the deployment of ransomware across exploited networks. These threat actors will possibly attempt to spread more quickly throughout the environment which requires threat hunters to continuously monitor for any tactics and techniques associated with typical ransomware threats. Every organization that has identified a vulnerable asset on their network should consider conducting persistent hunt operations on the previously exploitable networks. 

Below are long-term steps to take to help mitigate the lasting impacts of this vulnerability as they relate to implementing persistent hunt.

• Continue to Monitor Java and LDAP logs – As the vulnerability relies on JNDI to execute, and the associated LDAP call, continuous monitoring of these logs should be the priority for organizations.
• Hypothesis Exploitation as an Attack Vector – Multiple sources have shared observations of attackers using the Log4j vulnerability as an attack vector for dropping malware. Vx-underground has established a site with examples of malware seen delivered using Log4j and a quick glance will show some familiar names such as Mirai. SigmaHQ offers platform agnostic detection logic maintained by the cybersecurity community. The SANS Internet Storm Center provides a way to view honeypot data related to Log4j exploits which can inform how to hunt. As alerts begin to filter in for newly seen behavior, root cause analysis should consider a vulnerable Java application.
• Find All Running Java Instances – Hunting for Java processes may indicate a vector for attack. Using ‘jps’ commands, the Java Virtual Machine Process Status Tool will list all running Java processes by Process Identifier (PID) which should be verified against known and trusted processes.
• Monitor for Cobalt Strike Known Indicators of Compromise – Microsoft has reported seeing activity associated with Cobalt Strike (CS) beacons. Conducting hunts using known CS beacons can help exploited systems.

As many have stated, the effects of this vulnerability will be long term and require careful attention from security teams as adversaries are able to craft new ways to exploit vulnerable systems that defy previous countermeasures. 

Although Log4j is the prioritized critical vulnerability of the moment, organizations should be cautious about rotating all their resources to just addressing this singular issue. An attacker can leverage the noise being created with a publicized flaw and exploit other known vulnerabilities if security operations monitoring, detection, and analysis has been put on lower priority, overlooking the hot issue of the day. Security operations should be comprehensive and holistic of all vulnerabilities and threats to the organization, and this should not be stopped despite this current vulnerability.

This blog series is brought to you by Booz Allen DarkLabs. Our DarkLabs is an elite team of security researchers, penetration testers, reverse engineers, network analysts, and data scientists, dedicated to stopping cyber attacks before they occur.

This article is for informational purposes only; its content may be based on employees’ independent research and does not represent the position or opinion of Booz Allen. Furthermore, Booz Allen disclaims all warranties in the article's content, does not recommend/endorse any third-party products referenced therein, and any reliance and use of the article is at the reader’s sole discretion and risk.