On Friday, a lone Microsoft developer rocked the world when he revealed a backdoor had been intentionally planted in xz Utils, an open source data compression utility available on almost all installations of Linux and other Unix-like operating systems. The person or people behind this project likely spent years on it. They were likely very close to seeing the backdoor update merged into Debian and Red Hat, the two biggest distributions of Linux, when an eagle-eyed software developer spotted something fishy.
“This might be the best executed supply chain attack we’ve seen described in the open, and it’s a nightmare scenario: malicious, competent, authorized upstream in a widely used library,” software and cryptography engineer Filippo Valsorda said of the effort, which came frightfully close to succeeding.
Researchers have spent the weekend gathering clues. Here’s what we know so far.
XZ Utils is an essential component in practically every Linux-based system. This tool provides an extremely important role in lossless data compression and can be found in Unix-like systems globally. One critical aspect of XZ Utils is its support for the .lzma format, further emphasizing its importance.
Andres Freund, a developer for Microsoft’s PostgreSQL products, was recently faced with performance issues on a Debian system dealing with SSH, an essential protocol used extensively for remote logins. The main issue was that SSH logins were utilizing too much CPU and causing errors with valgrind, a utility for detecting memory management issues.
By way of close inspection and a stroke of luck, Freund found that updates to XZ Utils were causing the issues. Freund communicated to the Open Source Security List that these updates were not accidental; a backdoor had been intentionally added to the compression software.
The extent and complexity of the social engineering used for this backdoor is difficult to fully comprehend. Microsoft researcher, Thomas Roccia, posted a diagram on Mastodon to illustrate the extent of the almost successful backdoor planting operation. The scale of this backdoor would have made the 2020 SolarWinds event look minor in comparison.
Malicious code added to xz Utils versions 5.6.0 and 5.6.1 modified the way the software functions. The backdoor manipulated sshd, the executable file used to make remote SSH connections. Anyone in possession of a predetermined encryption key could stash any code of their choice in an SSH login certificate, upload it, and execute it on the backdoored device. No one has actually seen code uploaded, so it’s not known what code the attacker planned to run. In theory, the code could allow for just about anything, including stealing encryption keys or installing malware.
Any library can tamper with the inner workings of any executable it is linked against. Often, the developer of the executable will establish a link to a library that’s needed for it to work properly. OpenSSH, the most popular sshd implementation, doesn’t link the liblzma library, but Debian and many other Linux distributions add a patch to link sshd to systemd, a program that loads a variety of services during the system bootup. Systemd, in turn, links to liblzma, and this allows XZ Utils to exert control over sshd.
It would appear that this backdoor was years in the making. In 2021, someone with the username JiaT75 made their first known commit to an open source project. In retrospect, the change to the libarchive project is suspicious, because it replaced the safe_fprint funcion with a variant that has long been recognized as less secure. No one noticed at the time.
This story originally appeared on Ars Technica, a trusted source for technology news, tech policy analysis, reviews, and more. Ars is owned by WIRED’s parent company, Condé Nast.
In the subsequent year, JiaT75 proposed a patch to the XZ Utils mailing list. Almost instantly, a previously unseen participant, known as Jigar Kumar, entered the conversation asserting that the longtime XZ Utils maintainer, Lasse Collin, was not updating the software frequently or quickly enough. Kumar, backed by Dennis Ens and a few other newcomers to the list, persuaded Collin to recruit an additional developer for the project.
In January 2023, JiaT75 made their first commit to XZ Utils. Over the following months, JiaT75, who adopted the name Jia Tan, became increasingly active in XZ Utils. Tan, for instance, replaced Collins’ contact information with their own on oss-fuzz, a project scanning open source software for exploitable vulnerabilities. Tan also requested that oss-fuzz disable the ifunc function during testing, a modification that hindered it from detecting the malintent changes Tan would soon implement into XZ Utils.
In February of the present year, Tan established commits for versions 5.6.0 and 5.6.1 of XZ Utils. These updates incorporated the backdoor. In the ensuing weeks, Tan or others petitioned Ubuntu, Red Hat, and Debian developers to integrate the updates into their operating systems. Eventually, one of the two updates found its way into numerous releases, as reported by security firm Tenable. More details about Tan and the timeline can be found here.
To sum up, it enables someone possessing the correct private key to hijack sshd, the executable file in charge of establishing SSH connections, and thereafter execute malicious commands. The backdoor comes in the form of a five-stage loader employing a series of straightforward yet shrewd methods to conceal itself. It also facilitates the delivery of new payloads without necessitating major alterations.
Multiple people who have reverse-engineered the updates have much more to say about the backdoor. Developer Sam James provided an overview here.
Matt Burgess
R Douglas Fields
Kate O’Flaherty
Leif Wenar
In an online interview, developer and reverse engineer HD Moore confirmed the Sam James suspicion that the backdoor targeted either Debian or Red Hat distributions.
“The attack was sneaky in that it only did the final steps of the backdoor if you were building the library on amd64 (intel x86 64-bit) and were building a Debian or a RPM package (instead of using it for a local installation),” he wrote.
Paraphrasing observations from researchers who collectively spent the weekend analyzing the malicious updates, he continued:
During the process of SSH public key verification, the public key is checked against a specific fingerprint function. The contents of the key are decrypted using a pre-shared key before the public key is truly verified if it matches perfectly. The decrypted contents are then directly sent to the system.
If the fingerprint does not match, or if the decrypted contents do not fit into a specific format, the system reverts to the normal process of key verification without any signs of change.
The insertion of the backdoor is incredibly subtle. It capitalizes on a relatively unknown feature of the glibc to hook a function. This action is only instigated when the affected library gets loaded by a /usr/bin/sshd process on one of the affected distributions. Although there may be several other backdoors, the most discussed one employs function indirection to set the hook. The shellcode executes the payload, which was encoded into counterfeit xz test files. The SSH RSA key verification code is thus altered, allowing a magic public key received during regular authentication to give the attacker access.
this was their elaborate plan:
Researchers from networking firm Akamai also explain well how the backdoor works here.
At the moment, extremely little, especially for someone entrusted to steward a piece of software as ubiquitous and as sensitive as XZ Utils. This developer persona has touched dozens of other pieces of open source software in the past few years. At the moment, it’s unknown if there was ever a real-world person behind this username or if Jia Tan is a completely fabricated individual.
Additional technical analysis is available from the above Bluesky thread from Valsorda, researcher Kevin Beaumont, and Freund’s Friday disclosure.
Yes, it’s CVE-2024-3094.
There are several ways. One is this page from security firm Binarly. The tool detects implementation of IFUNC and is based on behavioral analysis. It can automatically detect invariants in the event a similar backdoor is implanted elsewhere.
There’s also a project called xzbot. It provides the following:
This story originally appeared on Ars Technica.
Updated 4/2/2024, 1:23 pm ET to include additional details.
