Using IFUNC to patch sshd was kind of elegant, it achieved rootkit like behaviour with a pre-existing mechanism. And sure, it might be possible for a secure daemon like sshd to drop enough privileges that it could protect itself from a malicious dynamically linked library.
But IFUNC was not required, neither was systemd. The game was lost as soon as the attacker had arbitrary code installed in a semi-common library. It doesn't have to get linked directly with sshd, it only needed to be linked into any program running as root, at least one time.
Most programs make zero effort to sandbox themselves, and as soon as one of those links with the malicious library, it could do anything. Like indirectly targeting sshd by patching its binary on disk (optionally hiding it with a rootkit), or using debug APIs to patch sshd in memory.
IFUNC, systemd, and the patched openssh are all irrelevant to the issue, that was simply the route this attacker took to leverage their foothold in libxz. There are thousands of potential routes the attacker could have taken, and we simply can't defend from all of them.
> Most programs make zero effort to sandbox themselves, and as soon as one of those links with the malicious library, it could do anything. Like indirectly targeting sshd by patching its binary on disk (optionally hiding it with a rootkit), or using debug APIs to patch sshd in memory.
I do not understand how you even expected sshd to sandbox itself. Its entire purposes is to (a) daemonize , (b) allow incoming connections in and then (c) forward (possibly-root) shell statements. All 3 things are 100% required for sshd and would have already allowed an attack like this. Any talk about sandboxing here is wishful thinking.
I recently tried to make something properly sandboxed and, my goodness, we have basically crafted an ecosystem where everything needs access to everything. No wonder docker, despite all it's faults, is how everyone does it. You need an entire linux distro completely accessible in your sandbox.
It was not essential to the exploit, but that does not mean it was irrelevant. More commonly used libraries are watched harder. The exploit was made much, much, worse by its indirect use by way of systemd. Approximately nobody wanted that feature and it still went in. That's something we need to be able to discuss.
OpenBSD exposes pledge() and unveil(), which allow programs to only access things they declare they need. So, even if the running SSH process gets exploited, it can't do anything the user it's running as can't do. sshd afaik runs as a root process which after authentication forks into another process, running as the target user.
There is always selinux if we want to add protection against arbitrary code running as root. Just because something operate as root does not mean it must have privileged access to everything.
1) IFUNC is hardly the only way to run code before main.
2) The alternative they present is arguably less secure because the function pointer will remain writable for the life of the process, whereas with IFUNC the GOT will eventually be made immutable (or can be... not sure if that's the default behavior). In general function pointers aren't great for security unless you explicitly make the memory backing the pointer(s) unwritable, which at least is easier to do for a global table than it is for things like C++ vtables (because there's the extra indirection through data pointers involved to get to the table).
> The alternative they present is arguably less secure because the function pointer will remain writable for the life of the process
They also suggest an alternative to storing the function pointer, store the bit flags that decide which function to call. That restricts the call targets to only the legitimate ones intended.
Yeah, this blog is misguided. As a higher level criticism: it's confusing[1] the technical details with the payload with the exploit chain that deployed it.
The interesting thing is obviously not that you can get code to run at high privilege level by modifying a system component. I mean, duh, as it were.
The interesting thing is that the attackers (almost) got downstream Linux distros to suck down and deploy that malicious component for them. And that's not down to an oddball glibc feature, it happened because they got some human beings to trust a malicious actor. GNU glibc can't patch that!
> The alternative they present is arguably less secure because the function pointer will remain writable for the life of the process
The article mentions this, and also points to mprotect which you can use to protect the pointer.
Why people jump to criticize without reading first? BTW, you can ask an LLM to check your critique, before posting, if you don't want to read the text.
Yes, liblzma could have used multiple routes to take over sshd. Once you're running inside the process it's game over. The exact details, like how they used ifunc and an audit hook, are very interesting, but ultimately not that important.
Some solaris engineer got a little too clever and decided that the modular part of the the auth system needed to be dynamic libs. Now it's all in one process space, hard to understand, hard to debug and fragile.
I really like openbsd's bsdauth, I don't know if it is actually any better than pam but because it is moduler at the process level it is possible for mere mortals to debug and make custom auth plugins. Sadly only obsd actually uses it. https://man.openbsd.org/login.conf.5#AUTHENTICATION
IFUNC should be implemented by software itself,
like switching functions on runtime/compile checks.
Why bother having a slower, insecure version that is less
flexible than a function pointer? I have to agree with author.
Glibc is filled with even more nasty hacks ripe for new exploits.
I agree so much and wished this was the main focus of the debate. It's more a question of why does this exist in the first place and not of how did they abuse it. Building only from source is the minimum required transparency and a CI/CD pipeline able to manipulate the artifact before release takes this away. I remember the outrage, when serde (i think it was) wanted to ship parts as pre-compiled binaries for build performance reasons...
Less indirection means faster code. If the dynamic loader is already using a level of indirection and you patch into that same indirection instead of adding another, you're not making it slower.
Yeah, that suggestion made me roll my eyes. It's the wrong granularity, there's no build system support, it's inconvenient (executable wrappers? require the user to understand all transitive deps?).
It also fails to mention glibc-hwcaps, which would've been a cleaner solution in the context.
The entire argumentation here is ridiculous. There's a big jump from "IFUNC slightly undermines RELRO" to "IFUNC is the real culprit". You could have gotten all but the same effect spawning a thread from a plain init or C++ constructor. No one should think that any relro, r^x or aslr or anything like this is going to deter anyone who can literally control the contents of the libraries which are linked in. They could, literally, exec a copy of sshd with a patched config if necessary.
I think I triggered the posting of this article in a reply elsewhere, to which you gave the same comment. And dammit, you're totally right, there's lots of other ways to stick arbitrary code into the initialization process at load time. Maybe the real problem is linking dependencies by shoving them into the same address space as whatever is using them and calling it a day. Memory-safe languages can help protect that model against accident, but it still takes just one evil insider to steal the keys to the kingdom.
Debian did not link OpenSSH with a 1.5 million-line library, because one doesn't exist. The library is libsystemd, which is comparatively tiny, and it is tiny so that sane things like Type=notify services get supported in more places with less pushback.
Yes, it could be smaller, broken up to remove compression support [0], what have you. But you should criticize the things that are actually problems, not some made-up bullshit about the whole of systemd being linked into everything that talks to it.
> Great. TFA's author thinks he cherry picked a sentence to make the project look bad.
Err... What? It's just a factual, non-judgemental description. Unlike your comment, which goes out of its way to call systemd names for whatever reason. Which just makes me less interested in what you have to say. Most people who rely on appeal to emotion to that extent are not in the right.
> systemd is a monstrous codebase and there lies shitload of exploits in it. Either intentional or accidental.
And yet...
1. practically all hyperscalers use it
2. desktops
3. container images, that power everything from docker to kubernetes use it
It helps that it's actively maintained, battle-tested as hell, and widely audited.
Point being, it's fun to hate on systemd, and maybe even hipster-like, and systemd is hardly perfect... but you are probably more likely to be exploited by a pypi or npm supply-chain attack.
> It helps that it's actively maintained, battle-tested as hell, and widely audited.
Is it actually audited? Or is it like OpenSSL... everybody uses it, but nobody looks under the hood cause it's gross in there? (Or well, nobody looked before Heartbleed anyway)
And it runs as PID1 on many distros and these are folks like RHEL, who have a huge interest in keeping it secure.
Pypi has an almost daily exploit announced in common and popular libraries, simply because the dependency graph is so huge. And this is in things that are almost certainly deliberately and by design exposed to insecure user input.
Again, it’s fun to hate on systemd, but in reality you are much more likely to be exploited by something else.
> Point being, it's fun to hate on systemd, and maybe even hipster-like, and systemd is hardly perfect... but you are probably more likely to be exploited by a pypi or npm supply-chain attack.
Can you even imagine pypi or npm compromising ssh this way?
> Can you even imagine pypi or npm compromising ssh this way?
Is ssh somehow sacrosanct in a way that any other RCE or credential stealing attack is different?
I don’t even know the last time I exposed ssh to the open internet.
But the fact with npm or pypi you can be exploited just by running the software you’ve already installed because the dependencies are everywhere on your system?
> Is ssh somehow sacrosanct in a way that any other RCE or credential stealing attack is different?
I see ssh as a very fundamental part of the system - in BSD terms it's in base not ports. Random packages from npm or pypi, sure, if you installed some slop off the internet and got exploited that's not so surprising. (Even those package managers themselves are not part of the base system, much less anything you install with them). But ssh should be safe!
> The short answer is that they have to. OpenSSH is developed by the OpenBSD community, for the OpenBSD community, and they do not give a flying Fedora about Linux.
What complete horseshit. I stopped reading there.
The OpenSSH Portable branch is maintained by OpenBSD developers and SystemD is a completely optional add-on so why on earth would they make it a dependency? If they didn't care about the Linux community they wouldn't develop this software *for free* for them. They can go write their own GNU SSH then.
It certainly doesn't help that there are 165+ definitions of what constitutes a "complete GNU+Linux system" some of which use SystemD and some which vow never to.
It's not the OpenBSD developers' fault some Linux distros use overly complex plumbing and can't agree on one standard for their OS unlike every other OS out there, including Windows.
The xz backdoor was a Debian and Red Hat issue because they maintained patches to fix problems of their own creation. No one else was affected. Why should the OpenBSD people care? It's not their problem.
The OP agrees with you... if you continue reading, they wrote
> These patches never went into Portable OpenSSH, because the Portable OpenSSH folks were ["not interested in taking a dependency on libsystemd"](link). And they never went into upstream OpenSSH, because OpenBSD doesn't have any need to support SystemD.
The language may have been harsher than it needed to and therefore could be more easily misunderstood, but I believe you are actually in agreement with them
It makes it sound even worse, cherry picking language like "not interested" as if the OpenBSD folks should shoulder blame for not being altruistic enough.
It reeks of trashing your benefactor, who gave you well-written free software, which you then made insecure with your own patches.
If you remove the roof of your car with a chainsaw and are inevitably injured later, is it the car manufacturer's fault they didn't offer that model as a convertible from the factory?
The better question is why are people still trying to assign blame all these years later? The IT world dodged a bullet but has moved on (and likely didn't learn from their mistakes as supply chain attacks are steadily increasing).
Okay. You could see it that way. Or you could read what the author wrote about who is to blame:
> No one person or team really made a mistake here, but with the benefit of hindsight it's clear the attackers perceived that the left hand of Debian/Fedora SSH did not know what the right hand of xz-utils was doing.
I guess it's up to interpretation, but I read it the complete opposite way, as in Linux distributions should not think so highly of themselves as to expect OpenBSD to conform and adapt to their mess, and OpenBSD rightfully should not be expected to "give a flying Fedora about Linux".
>Did the OpenSSH folks know (or care) that ifunc was a thing? It's certainly not a thing on OpenBSD.
I do not know why you were down-voted, maybe you deserved no up-votes, but down-votes to me were a bit extreme :) But that quote tends to indicate to me the author put a little blame on OpenSSH Developers. Maybe the author did not intend it to be read in the way I read it.
OpenSSH developers should not need to know what or why systemd distros apply patches to OpenSSH, the distro I use, Slackware, did not have this vulnerability because the Slackware team, AFAIK, only adds patches if the package does not compile. If other distros did that this issue would not have occurred.
To me the issue was patching OpenSSH for some systemd thing. Maybe IFUNC was part of the issue, but the real issue was patching OpenSSH.
But I know one thing, I never heard of IFUNC and after reading about it, I will avoid that as much as I can. So at least I was educated :)
Using IFUNC to patch sshd was kind of elegant, it achieved rootkit like behaviour with a pre-existing mechanism. And sure, it might be possible for a secure daemon like sshd to drop enough privileges that it could protect itself from a malicious dynamically linked library.
But IFUNC was not required, neither was systemd. The game was lost as soon as the attacker had arbitrary code installed in a semi-common library. It doesn't have to get linked directly with sshd, it only needed to be linked into any program running as root, at least one time.
Most programs make zero effort to sandbox themselves, and as soon as one of those links with the malicious library, it could do anything. Like indirectly targeting sshd by patching its binary on disk (optionally hiding it with a rootkit), or using debug APIs to patch sshd in memory.
IFUNC, systemd, and the patched openssh are all irrelevant to the issue, that was simply the route this attacker took to leverage their foothold in libxz. There are thousands of potential routes the attacker could have taken, and we simply can't defend from all of them.
I do not understand how you even expected sshd to sandbox itself. Its entire purposes is to (a) daemonize , (b) allow incoming connections in and then (c) forward (possibly-root) shell statements. All 3 things are 100% required for sshd and would have already allowed an attack like this. Any talk about sandboxing here is wishful thinking.
2) The alternative they present is arguably less secure because the function pointer will remain writable for the life of the process, whereas with IFUNC the GOT will eventually be made immutable (or can be... not sure if that's the default behavior). In general function pointers aren't great for security unless you explicitly make the memory backing the pointer(s) unwritable, which at least is easier to do for a global table than it is for things like C++ vtables (because there's the extra indirection through data pointers involved to get to the table).
They also suggest an alternative to storing the function pointer, store the bit flags that decide which function to call. That restricts the call targets to only the legitimate ones intended.
The interesting thing is obviously not that you can get code to run at high privilege level by modifying a system component. I mean, duh, as it were.
The interesting thing is that the attackers (almost) got downstream Linux distros to suck down and deploy that malicious component for them. And that's not down to an oddball glibc feature, it happened because they got some human beings to trust a malicious actor. GNU glibc can't patch that!
[1] Incorrectly, as you point out.
The article mentions this, and also points to mprotect which you can use to protect the pointer.
Why people jump to criticize without reading first? BTW, you can ask an LLM to check your critique, before posting, if you don't want to read the text.
1. It impiles some function pointers to be writable temporarily, not all of them.
2. It doesn't hide writable pointers from a cursory glance not familiar with IFUNC.
Some solaris engineer got a little too clever and decided that the modular part of the the auth system needed to be dynamic libs. Now it's all in one process space, hard to understand, hard to debug and fragile.
I really like openbsd's bsdauth, I don't know if it is actually any better than pam but because it is moduler at the process level it is possible for mere mortals to debug and make custom auth plugins. Sadly only obsd actually uses it. https://man.openbsd.org/login.conf.5#AUTHENTICATION
It also fails to mention glibc-hwcaps, which would've been a cleaner solution in the context.
The title is just clickbait.
As I know C++ allows running arbitrary code before main too - for constructors of global variables. Does it bring security risks too?
(In practice _start calls __libc_start_main, a libc function that handles all of that).
Curses, thwarted again!
Yes, it could be smaller, broken up to remove compression support [0], what have you. But you should criticize the things that are actually problems, not some made-up bullshit about the whole of systemd being linked into everything that talks to it.
0: https://github.com/systemd/systemd/issues/32028
Err... What? It's just a factual, non-judgemental description. Unlike your comment, which goes out of its way to call systemd names for whatever reason. Which just makes me less interested in what you have to say. Most people who rely on appeal to emotion to that extent are not in the right.
And yet...
1. practically all hyperscalers use it
2. desktops
3. container images, that power everything from docker to kubernetes use it
It helps that it's actively maintained, battle-tested as hell, and widely audited.
Point being, it's fun to hate on systemd, and maybe even hipster-like, and systemd is hardly perfect... but you are probably more likely to be exploited by a pypi or npm supply-chain attack.
Is it actually audited? Or is it like OpenSSL... everybody uses it, but nobody looks under the hood cause it's gross in there? (Or well, nobody looked before Heartbleed anyway)
This is 2026, not 2014 when heartbleed came out.
And it runs as PID1 on many distros and these are folks like RHEL, who have a huge interest in keeping it secure.
Pypi has an almost daily exploit announced in common and popular libraries, simply because the dependency graph is so huge. And this is in things that are almost certainly deliberately and by design exposed to insecure user input.
Again, it’s fun to hate on systemd, but in reality you are much more likely to be exploited by something else.
Can you even imagine pypi or npm compromising ssh this way?
Is ssh somehow sacrosanct in a way that any other RCE or credential stealing attack is different?
I don’t even know the last time I exposed ssh to the open internet.
But the fact with npm or pypi you can be exploited just by running the software you’ve already installed because the dependencies are everywhere on your system?
I see ssh as a very fundamental part of the system - in BSD terms it's in base not ports. Random packages from npm or pypi, sure, if you installed some slop off the internet and got exploited that's not so surprising. (Even those package managers themselves are not part of the base system, much less anything you install with them). But ssh should be safe!
> The short answer is that they have to. OpenSSH is developed by the OpenBSD community, for the OpenBSD community, and they do not give a flying Fedora about Linux.
What complete horseshit. I stopped reading there.
The OpenSSH Portable branch is maintained by OpenBSD developers and SystemD is a completely optional add-on so why on earth would they make it a dependency? If they didn't care about the Linux community they wouldn't develop this software *for free* for them. They can go write their own GNU SSH then.
It certainly doesn't help that there are 165+ definitions of what constitutes a "complete GNU+Linux system" some of which use SystemD and some which vow never to.
It's not the OpenBSD developers' fault some Linux distros use overly complex plumbing and can't agree on one standard for their OS unlike every other OS out there, including Windows.
The xz backdoor was a Debian and Red Hat issue because they maintained patches to fix problems of their own creation. No one else was affected. Why should the OpenBSD people care? It's not their problem.
> These patches never went into Portable OpenSSH, because the Portable OpenSSH folks were ["not interested in taking a dependency on libsystemd"](link). And they never went into upstream OpenSSH, because OpenBSD doesn't have any need to support SystemD.
The language may have been harsher than it needed to and therefore could be more easily misunderstood, but I believe you are actually in agreement with them
It reeks of trashing your benefactor, who gave you well-written free software, which you then made insecure with your own patches.
If you remove the roof of your car with a chainsaw and are inevitably injured later, is it the car manufacturer's fault they didn't offer that model as a convertible from the factory?
The better question is why are people still trying to assign blame all these years later? The IT world dodged a bullet but has moved on (and likely didn't learn from their mistakes as supply chain attacks are steadily increasing).
> No one person or team really made a mistake here, but with the benefit of hindsight it's clear the attackers perceived that the left hand of Debian/Fedora SSH did not know what the right hand of xz-utils was doing.
with OpenBSD not even being mentioned here
I do not know why you were down-voted, maybe you deserved no up-votes, but down-votes to me were a bit extreme :) But that quote tends to indicate to me the author put a little blame on OpenSSH Developers. Maybe the author did not intend it to be read in the way I read it.
OpenSSH developers should not need to know what or why systemd distros apply patches to OpenSSH, the distro I use, Slackware, did not have this vulnerability because the Slackware team, AFAIK, only adds patches if the package does not compile. If other distros did that this issue would not have occurred.
To me the issue was patching OpenSSH for some systemd thing. Maybe IFUNC was part of the issue, but the real issue was patching OpenSSH.
But I know one thing, I never heard of IFUNC and after reading about it, I will avoid that as much as I can. So at least I was educated :)