By Andy Greenberg and Matt Burgess
The scourge of software supply chain attacks—an increasingly common hacking technique that hides malicious code in a widely used legitimate program—can take many forms. Hackers can penetrate an update server to seed out their malware, or even break into the network where the software was developed to corrupt it at the source. Or, in the case of one particularly insidious software supply chain attacker known as Jia Tan, they can spend two years politely and enthusiastically volunteering to help.
Over the weekend, the cybersecurity and open source software community was shocked by the news that a relatively new, experimental version of XZ Utils—a compression utility integrated into many popular distributions of Linux—contained a backdoor that would have allowed hackers in possession of a specific private key to connect to the backdoored system and run their own commands as an administrator. Only some chance detective work carried out by a lone Microsoft engineer, Andres Freund—who’d detected a strange delay in how the remote connection protocol SSH was running in a version of the Linux variant Debian—caught the spy trick before it ended up in many millions of systems worldwide.
That XZ Utils backdoor, it’s now clear, was inserted by none other than the lead open source steward of XZ Utils, a developer who went by the name Jia Tan. In the wake of the backdoor’s discovery, a mystery percolating through the tech world remains: Who is Jia Tan, and who did he, or she—or very likely they—truly work for?
Jia Tan exploited open source software’s crowdsourced approach to coding whereby anyone can suggest changes to a program on code repositories like GitHub, where the changes are reviewed by other coders before they’re integrated into the software. Peeling back Jia Tan’s documented history in the open source programming world reveals that they first appeared in November 2021 with the GitHub username JiaT75, then made contributions to other open source projects using the name Jia Tan, or sometimes Jia Cheong Tan, for more than a year before beginning to submit changes to XZ Utils.
By January 2023, Jia Tan’s code was being integrated into XZ Utils. Over the next year, they would largely take control of the project from its original maintainer, Lasse Collin, a change driven in part by nagging emails sent to Collin by a handful users complaining about slow updates. (Whether those users were unwitting accomplices, or actually working with Jia Tan to persuade Collin to relinquish control, remains unclear. None of the users replied to requests for comment from WIRED.) Finally, Jia Tan added their stealthy backdoor to a version of XZ Utils in February of this year.
That inhumanly patient approach, along with the technical features and sophistication of the backdoor itself, has led many in the cybersecurity world to believe that Jia Tan must, in fact, be a handle operated by state-sponsored hackers—and very good ones. “This multiyear operation was very cunning, and the implanted backdoor is incredibly deceptive,” says Costin Raiu, who until last year served as the most senior researcher and head of the global research and analysis team at Russian cybersecurity firm Kaspersky. “I’d say this is a nation-state-backed group, one with long-term goals in mind that affords to invest into multiyear infiltration of open source projects.”
Dell Cameron
Matt Burgess
Julian Chokkattu
Caroline Haskins
As for which nation, Raiu names the usual suspects: China, Russia, and North Korea. He says it’s still too early to know the true culprit. “One thing is for sure clear,” he adds. “This was more cunning than all previous software supply chain attacks I’ve seen.”
As scrutiny around Jia Tan has mounted since the revelation of the XZ Utils backdoor last Friday, researchers have noted that the persona has remarkably good operational security. Independent security reporter Brian Krebs writes that he could find “zero trace” of Jia Tan’s email address outside of the messages they sent to fellow open source contributors, even after scouring breached databases. Jia Tan also appears to have routed all their communications through a VPN with a Singaporean IP address.
The lack of any other online presence linked to Jia Tan points toward the account being a “single-purpose invented persona” and indicates how much sophistication, patience, and thought was put into developing the backdoor, says Will Thomas, an instructor at the SANS Institute, a cybersecurity training firm. The Jia Tan persona has vanished since the backdoor was discovered, and emails sent by WIRED to a Gmail address linked to it have gone unanswered. Jia Tan’s GitHub account has been suspended, a company spokesperson tells WIRED.
In fact, the only real footprints Jia Tan appears to have left behind were their contributions to the open source development community, where they were a prolific contributor: Disturbingly, Jia Tan’s first code change was to the “libarchive” compression library, another very widely used open source component. That first change swapped one function with a less secure alternative, potentially attempting another malicious code change, notes developer Evan Boehs in his detailed Jia Tan timeline—though the problem has since been fixed.
In total, Jia Tan made 6,000 code changes to at least seven projects between 2021 and February 2024, according to Michael Scott, the cofounder of the cybersecurity firm NetRise who previously worked in the Marine Corps cyberwarfare group under US Cyber Command. Determining all the branching effects of those changes is nearly impossible, Scott says. Because those changes, known as “commits,” are often batched into collections in a process known as “squashing commits,” it’s not always apparent which exact changes were made by Jia Tan. And the difficulty of tracing which of the many versions of a library like libarchive ended up in which software adds yet another layer of obfuscation. “It’s going to be a bit of a mess pulling on this thread and trying to figure out where all these things ended up,” Scott says.
Scott notes that, throughout this time, Jia Tan was also emailing with other contributors, writing in a “very concise, very dry,” but not unfriendly tone that Scott compares to the output of ChatGPT. “Nice job to both of you for getting this feature as far as it is already,” Jia Tan wrote at one point. Or, at another: “Let me know your thoughts on these patches when you have a chance :)” Jordi Mas, a developer who contributed to XZ Utils and had emailed “feedback” from Jia Tan, says in retrospect that the account went to extra levels to build trust in the persona.
Dell Cameron
Matt Burgess
Julian Chokkattu
Caroline Haskins
Ultimately, Scott argues that those three years of code changes and polite emails were likely not spent sabotaging multiple software projects, but rather building up a history of credibility in preparation for the sabotage of XZ Utils specifically—and potentially other projects in the future. “He just never got to that step because we got lucky and found his stuff,” says Scott. “So that’s burned now, and he’s gonna have to go back to square one.”
Despite Jia Tan’s persona as a single individual, their yearslong preparation is a hallmark of a well-organized state-sponsored hacker group, argues Raiu, the former Kaspersky lead researcher. So too are the technical hallmarks of the XZ Utils malicious code that Jia Tan added. Raiu notes that, at a glance, the code truly looks like a compression tool. “It’s written in a very subversive manner,” he says. It’s also a “passive” backdoor, Raiu says, so it wouldn’t reach out to a command-and-control server that might help identify the backdoor’s operator. Instead, it waits for the operator to connect to the target machine via SSH and authenticate with a private key—one generated with a particularly strong cryptographic function known as ED448.
The meticulous design of the backdoor could be an act of US hackers, suggests Raiu, but this is rather improbable as the US usually avoids sabotaging open source projects, and in the event they did, the National Security Agency would likely resort to a quantum-resistant cryptographic function, which ED448 does not provide. This then leaves non-US groups that are known for supply chain attacks, as Raiu suggests, like China’s APT41, North Korea’s Lazarus Group, and Russia’s APT29.
Jia Tan, at a first glance, clearly appears East Asian—or is intended to be so. The time zone for Jia Tan’s commits is UTC+8, which aligns with China’s zone, and is only an hour different from North Korea’s. However, an analysis by two researchers, Rhea Karty and Simon Henniger, suggests that Jia Tan possibly manipulated the time zone of their computer to UTC+8 before making each commit. Indeed, a few commits were made with a computer set to Eastern European or Middle Eastern time zones, perhaps when Jia Tan overlooked making the change.
“Another indication that they are not from China is that they worked on significant Chinese holidays,” assert Karty and Henniger, who are students at Dartmouth College and the Technical University of Munich, respectively. They observe that Jia Tan also didn’t contribute to new code on Christmas or New Year’s. Boehs, the developer, further propounds that the majority of the work commences at 9 am and concludes at 5 pm for Eastern European or Middle Eastern time zones. “The time range of the commits suggests that this was not a side project they were working on outside of their regular employment,” comments Boehs.
While these observations might point toward countries like Iran and Israel, the majority of clues lead back to Russia, and expressly Russia’s APT29 hacking group. This argument is posited by Dave Aitel, a former NSA hacker and founder of the cybersecurity firm Immunity. Aitel emphasizes that APT29—broadly assumed to be operative for Russia’s foreign intelligence agency, known as the SVR—exhibits a level of technical carefulness which is rare among other hacker groups. APT29 was also responsible for carrying out the Solar Winds compromise, which may arguably be the most skillfully laid out and successful software supply chain attack in history. This operation aligns more closely with the style of the XZ Utils backdoor than the more rudimentary supply chain attacks of APT41 or Lazarus.
“It could very well be someone else,” says Aitel. “But I mean, if you’re looking for the most sophisticated supply chain attacks on the planet, that’s going to be our dear friends at the SVR.”
Security researchers agree, at least, that it’s unlikely that Jia Tan is a real person, or even one person working alone. Instead, it seems clear that the persona was the online embodiment of a new tactic from a new, well-organized organization—a tactic that nearly worked. That means we should expect to see Jia Tan return by other names: seemingly polite and enthusiastic contributors to open source projects, hiding a government’s secret intentions in their code commits.
Updated 4/3/2024 at 12:30 pm ET to note the possibility of Israeli or Iranian involvement.
