XZ Backdoor (CVE-2024-3094) - A hidden backdoor in open-source software

On March 29th, Andres Freund (A Microsoft developer) created a thread in oss-security Openwall (a well-known mailing list for open-source security) which starts like this:

Hi, After observing a few odd symptoms around liblzma (part of the xz package) on Debian sid installations over the last weeks (logins with ssh taking a lot of CPU, valgrind errors) I figured out the answer:

The upstream xz repository and the xz tarballs have been backdoored.

At first I thought this was a compromise of debian's package, but it turns out to be upstream. ....

Basically, he noticed that the ssh login process was taking about 500ms more than usual. Then he started to analyze the root cause of the issue and he found out that xz (a popular library for lossless data compression widely used in unix-like systems) was compromised by an attacker who introduced a backdoor on it.

What is really interesting (and scary at the same time) is the way the malicious actor perpetrated the attack.

xz code (now disabled) was hosted on GitHub and it's completely open source (so anyone can contribute and the maintainer can accept or discard changes). However, the attacker (whose name is Jia Tan) started to commit to this repository since February 2022 (you can browse his activity on GitHub since the account is still there). He slowly gained the trust and credibility of the project maintainer, until 9th March 2024, when he used a clever technique to hide a backdoor by using 2 test files and several stages of payload's obfuscation. Also, one portion of the backdoor (which is executed in multiple stages) was only present in the tarballs of the affected versions (5.6.0 and 5.6.1) and it was not included in the source control version (so it was harder to find it).

The following image summarizes what happened from 2021 (GitHub account creation) until the final packaging of the backdoor inside xz/libzma (credits to Thomas Roccia for this amazing infographic):

Impact

The attacker can basically execute any arbitrary command (RCE - Remote Code Execution) on the affected systems that have an sshd daemon running and exposed. Only the attacker can execute commands because the payload must be signed with a Ed448 key only known by the attacker. More details here.

Fortunately, only few Linux distributions were affected by default (especially the ones with unstable/testing/development branches or the rolling-release distributions) among which Fedora (40,41), Debian (testing, unstable, experimental) and Arch Linux.

At the moment, we know that the payload works in some specific conditions:

• you must have an sshd daemon running
• sshd must be patched to support systemd-notify (which actually uses xz/liblzma as dependency)

Other attack scenario are possible and reverse engineers are still working to provide additional information.

If you want to check if you are using the backdoored xz library, run the following command:

strings `which xz` | grep '5\.6\.[01]'

A vulnerable system would show something like:

xz (XZ Utils) 5.6.1

and you should immediately downgrade to 5.4.6.

If you dont' get any output, you are safe.

More information here.

Final Thoughts

This security incident give us the opportunity to think about few important things.

1. Attacks are becoming more and more sophisticated. The way the malicious actor was able to gain reputation and credibility was insane. He acted with patience and he leveraged social engineering techniques to push the maintainer the to give him full control of the repository. Also he created fake accounts to push the infected xz-utils package to be included in debian (like here). Furthermore, he used advanced and pretty clever techniques to hide and obfuscate the payload. I personally believe that while software build mechanisms are essential, their complexity can inadvertently create opportunities for previously unknown or less feasible attack vectors to be exploited.

2. What does it mean to maintain an open-source project? The original maintainer of the project (Lasse Collin, who also created this page to provide updates about the incident) was completely overwhelmed by the requests coming from people using the xz library and complaining about the slow release of new features. In this thread Lasse also admitted to suffer "longterm mental health issues", and he was thinking to give Jia Tan (the guy who introduced the backdoor) a bigger role for the xz project. Probably we all need to understand what kind of effort is required to maintain an open-source project, especially considering that most of them do not receive any financial compensation despite being used everyday by thousands of multi-millionaire corporations.

3. Can we blindly trust open-source? We all know that software has bugs and bugs can create security issues, that can be introduced by developer's mistakes. This is normal and we mostly know how to deal with it (for example by adopting Software Composition Analysis tools and preventing vulnerable third-party dependencies from being imported in our application ecosystem as soon as a vulnerability has been discovered). However, this time is different: how do we trust contributors for an open-source project? While many contributors genuinely aim to improve the software and benefit the community, others may have ulterior motives, such as exploiting vulnerabilities for personal (?) gain. Considering that this security incident was identified by chance, how many others might exist that are still there and not yet identified?