The log4j vulnerability shows vulnerability management can’t come down to only a scan

The first article I saw on December 10 about the now infamous log4j remote code execution vulnerability (CVE-2021–44228), buried the lede. Posted at 10:39pm Central time on December 9, the headline read: “Minecraft and other apps face serious threat from Log4j code execution bug.”

Of course, the log4j Java library is widely used outside of Minecraft, including on things as boring as enterprise applications.

The announcement of this log4j vulnerability sent companies throughout the world scrambling to find copies of log4j anywhere in their internet-facing infrastructure: homegrown apps, appliances, third party software, VPNs—you name it. Less than 24 hours after the first post about this issue, attackers were already scanning the internet for vulnerable systems — and finding them easily.

Under the best of circumstances, many organizations have a difficult time pinpointing what exact software is loaded on their systems. Trying to find software on the fly is difficult, and, worse, network-based vulnerability scanners are not usually up for the job.

Still, the network-based scanner companies acted as fast as they could. Qualys had a detection by Friday night, and Nessus had one Saturday afternoon.

If companies were waiting for these detections to find log4j, they might have been disappointed when the detection was released. Network-based remote detection of an issue like log4j is often difficult — unless a header replies with specific version information — which log4j generally wouldn’t do.

Network-based scanners, when performing network or agent-based scans, typically check common locations on the filesystem or installed packages, like RPMs or absolute paths on the filesystem. For homegrown software, required libraries are often installed in the application directory, rather than the general filesystem. This means that such scanners won’t find log4j installs outside of certain expected locations, as Qualys notes in its own blog post about this vulnerability.

This is a problem not only for log4j, but many other binaries and libraries. There could be hundreds of pieces of software just as exploitable as log4j sitting on your servers, and, unless you do filesystem analysis, you wouldn’t know.

Interestingly, even when Qualys rolled out enhanced detection for Java versions, by searching the entire filesystem instead of only a few select paths, customers must have complained. How much did they complain? So much so that Qualys added a field to the results to indicate whether it was found in the “standard” location or an “enhanced” location. The Qualys forums are filled with requests to disable the new checks. More information, after all, means more work to do.

For those of us interested in actually finding vulnerable log4j or other software, though, what’s an upstanding vulnerability manager to do?

First, use your existing systems to your advantage. Check your build systems, they can provide you a list of software versions embedded in your homegrown apps.

Do you have a security tool that records all files on the filesystem (like Tripwire)? You can search through that to find log4j filenames. Or, if you lack such a system, crawl all your filesystems looking for log4j libraries. Sure, you’ll miss any that have been renamed, but you could find ones that you didn’t expect.

If you have a security team (or access to a consulting security team), they can create their own remote tests. Also, Web Application Scanners probably would be more likely to detect this issue more than a simple vulnerability scan would, if that’s something you have in your toolbelt.

To really solve the log4j issue, don’t only rely on network-based vulnerability scanners. You need to scan all libraries and binaries embedded in your homegrown applications. Invest in software analysis scanners like Synk and Prisma, or container scanning tools, which report on CVEs for libraries embedded in your homegrown apps. If even you can’t do it now, they can be your next step in maturing your program.

When you think you’ve resolved this incident, do a post-mortem. What information were you missing, and how can you start storing it now? This, after all, is just one of many vulnerabilities to come, and you need to be prepared for next time.

During this holiday season, please consider giving to your local food bank. In the Chicago area, I recommend the Greater Chicago Food Depository.

Based in Chicago, Jerry Galvin has over 15 years of experience in data center and cybersecurity operations. He currently specializes in vulnerability management. Contact him on Twitter at @jerry_galvin.