But, if ld.so supports $ORIGIN already, why does the kernel needs to support it also? Or rather, why can't the kernel leverage ld.so and do this entirely on userspace?
$ORIGIN was only supported when the loader was looking for other dependencies.
The loader itself (field PT_INTERP) is loaded by the kernel. So prior to this change, every program must hardcode the absolute path to ld.so.
With support for an $ORIGIN-relative loader, each program could use its own copy of ld.so.
Each program having its own loader is an anti pattern. Such a requirement is overkill. You can have a single loader that supports everything on the system.
Not with NixOS. ld.so is tied to a version of glibc, in ways that can be subtly incompatible. And nixos can have multiple glibc version installed on a single machine.
Besides, it allows for upgrades/downgrades to be done in a way that's much less error-prone.
In theory, ld.so could provide a stable interface to its dynamic loading capabilities independently of glibc. Then glibc would not have to be updated in concert with ld.so. There's no inherent reason that glibc should be the library shipping a dynamic linker - we could easily be in a world where libld was developed independently from libc.
On Windows, things are less modular, so it's less of an issue. That said, there's also weird shenanigans when it comes to the CRT (which can be statically linked) vs ntdll (which provides the actual linker implementation), that can make certain niche features of the linker misbehave (delay loading in particular is weird).
In practice, ld.so is a piece of glibc. If you want to build ld.so from scratch, you have to clone glibc or get a tarball of glibc, and configure and build glibc.
You could make the argument that the glibc developers should split ld.so into its own subprojects. What for? That would just bring the extra responsibility of making sure that variations in glibc version work with variations in ld.so version for whatever reason.
What would happen in practice is that they would keep their version numbers in lock step, and systems integrators would use the same version. While the upstream glibc has to go through a dance of pretending that someone cares about their independence.
still, eventually there's a need to support on one platform different ld.so/glibc pairs (even if they are API/ABI compatible)
it seems nixos could set up a wrapper that invokes the right ld.so based on the executable. though at this point they could probably edit the ELF binaries and patch the fixed path to ld.so when nix is installing the program.
yeah, it seems strange that this needs kernel support. but more eBFP extension points are usually welcome, so sure, why not?
> though at this point they could probably edit the ELF binaries and patch the fixed path to ld.so when nix is installing the program.
That's exactly what they're doing right now - though instead of being "when installing the program", it's "when compiling the program". The problem with hardcoding the path is that it pins the "nix store" (where nix installs all of its programs) to a hardcoded location. If you want to move it, you have to rebuild all your packages - which is suboptimal.
In theory, nix could have a system in place to just patch all binaries when moving the nix store, but that would be incompatible with content-addressed derivation and otherwise break some other nice properties of the nix store.
Not my area, but isn't it really only because glibc doesn't maintain stable interfaces across versions? If it did, you absolutely could use the same ld.so with different glibc versions. But it doesn't, so here we are.
The Windows situation is way different: every process is supposed to link against kernel32.dll, that's the public interface to the kernel. In Linux, glibc is just one of many C stdlib implementations, you can have many versions of glibc on the same system, etc.
Sure, but if you want dynamic loading from your c stdlib (which is defensible IMO), and you want the behavior/implementation to match with the loader, then you need some kind of coupling somewhere no?
You could have a very slim libdlopen that is used by both loader and libc, but I don't really see how that's any improvement/much different.
That's how Windows handles it - the OS libraries are language runtime agnostic and do not export language specific symbols nor need them, and language runtimes are distributed often separately.
So your C stdlib does not have dlopen() (in fact, it's not really part of C stdlib in POSIX anyway!) but you can link with standard OS-level ABI to OS-provided runtime services like "find a library and symbol from it" - AmigaOS did something similar.
$ /lib64/ld-linux-x86-64.so.2
/lib64/ld-linux-x86-64.so.2: missing program name
Try '/lib64/ld-linux-x86-64.so.2 --help' for more information.
!1!
$ /lib64/ld-linux-x86-64.so.2 --version
ld.so (GNU libc) stable release version 2.34.
^^^^^^^^^^^^^^^^
Copyright (C) 2021 Free Software Foundation, Inc.
This is free software; see the source for copying conditions.
There is NO warranty; not even for MERCHANTABILITY or FITNESS FOR A
PARTICULAR PURPOSE.
There is much more value to be had in making glibc properly backward compatible so you can have a single one that can be used with everything than trying to make it so that you can swap everything around, creating extra complexity and compatibility risks.
> […] in making glibc properly backward compatible […]
You could only do that going forward though, and would be stuck, at least for a time, with the historic versions that still need extra handling. In an ideal world that wouldn't be needed, but in an ideal world you'd not need multiple versions of glibc at all so we aren't there.
Support in the kernel means that it will work even if applied to old packages that for some other reason need a held-back version of glibc. As mentioned elsewhere it also gives the feature to #! directives in scripts too.
How is that more valuable? It comes with two big pitfalls:
- It would still not allow downgrades to work properly
- It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together
Meanwhile, having different ld.so has many good use-cases, like simplifying development of ld.so itself, allowing them to swapped during updates in ways that are safer, etc...
And the eBPF binfmt support is a rather simple, generic mechanism that is likely to have many other use-cases beyond ld.so. So it's overall a pretty good resolution to the issue?
>It would still not allow downgrades to work properly
If you really need this, then there are options like swapping the linker with an older version or making a hardcoded linker that now points to an older one.
>It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together
Every language runtime has to care about not breaking compatibility with apps that have been released already. This is not a new or unique problem.
Unfortunately, Glibc and loader are linked together intrinsically in the Linux ecosystem. So, if you want to be able to launch a program reliably, shipping your own loader and libc might actually be the only way.
Glibc is backward compatible though, they even have symbol versioning to provide multiple versions of the same symbol. So as long as you have the same or a newer version of glibc (than what was built against) you should be good to go. And I don't remember hearing about breakages for this.
Other libraries on the system is far more hit and miss, but glibc is quite compatible.
The other way around is harder though, you can't take a program built against a newer glibc and run it on an older version. So you generally need to spin up a container with some LTS distro and build your binary in it if you want it to be maximally compatible. (However, zig apparently is able to deal with this by shipping a mapping between glibc versions and symbol versions and doing the linking themselves. You can even use zig to link rust code using cargo-zigbuild and get that benefit.)
But if you want to be maximally portable: static linking against musl. Though beware that many things are slower in musl, such as the allocator.
Reading the linked bug report about executable stacks they fixed it? So they did the right thing. I'm not saying there will never be bugs (no non-trivial software is bug free), but as long as those are handled correctly that seems reasonable to me.
No, glibc isn't backwards compatible; I've had instances when the loader would refuse to load the executable because the installed glibc is too new for it.
> why can't the kernel leverage ld.so and do this entirely on userspace?
Because Linux is completely independent from its user space. There is no "the loader in Linux", it just happens that GNU wrote one and it's widely used.
You have the right idea but the wrong specifics. The boundary you're alluding to isn't the kernel/userspace boundary, it's the `libc` boundary, which is admittedly privileged by convention if not by Ring 0.
On most Linux systems this is regrettably `glibc` (in a container where you get to choose everyone chooses the superior option of `musl`), on all Darwin systems this is `libstandard`.
This thread is more concerned with Linux, so you are probably dealing with `/lib64/ld-linux-x86-64.so.2`, which operates in userspace but is by convention and opacity quite clearly part of "the system".
The kernel modification is a much bigger, much weirder side effect of a weird Nix loyalty test around shebang lines in shell scripts, on which is has opted to be intentionally and violently incompatible with everything for no benefit other than incompatibility.
That’s a weird rant which is manifestly false. On Nix the benefit you get from the incompatibility is the system can roll forward and back while it’s running so you can do big changes (eg updating or rolling back) the entire system without affecting running processes. You can also run groups of processes in isolation with different sets (or versions) of packages from the rest of the system.
It’s fair to argue about whether those things are really important or whether nix does this in the best way, but to claim that the only benefit is incompatibility is just obvious nonsense.
The privileged path is already there. `/usr/bin/env` is no different from `/bin/bash`, `/bin/sh`, or any other ELF artifact at a known place. The argument is made, the argument is spurious. It is made in the other direction regarding the driver run path, just as religious, opposite ruling from purity court. No one even knows why, the trail goes cold in a mysterious 2012 commit about Mesa, it's literally a performance to bring the plane cargo back. Receipts for claim: #141803.
Disagreeing with you doesn't make something a rant. If it's not clear how deep my Nix expertise is I will demonstrate it to any level you like: I'm just as entitled to an opinion as you are and I would appreciate it if the nixpkgs community was a little less rude to anyone who disagrees with some dogma no matter their knowledge. The nixpkgs community is not regarded as healthy or friendly after an internecine faction war that split three ways twice, producing four implementations none of which work, and if that reputation is ever going to mend, it will be because less than every Nix person has precisely zero chill the nanosecond anyone disagrees with them about anything.
It's no different other than it's a layer of indirection which allows bash to change in a way that would not be possible if scripts directly referenced /bin/bash or /usr/bin/bash. The only thing you can't change is the /usr/bin/env binary itself but that's not something that changes very much if at all. On my nix box, bash is at /run/current-system/sw/bin/bash and "current-system" is a symlink to a particular build in /nix/store. So /usr/bin/env is creating a layer of indirection so if I rebuild the system, the previous version can be kept around and I can roll back instantly at the boot menu and it will roll everything back to a consistent package set. Now as I say, you could achieve this in other ways eg fs snapshots etc. But to say that the change doesn't give you anything is just very obviously wrong.
And you don't need to worry- it's perfectly obvious how deep your Nix expertise is.
You've just used a personal attack to shout down someone who disagrees in good faith over a distinction you simultaneously describe in your own words as "something you could achieve other ways" (yeah, more than a little bit) referencing an artifact that "is a symlink", "on my nix box".
The only thing I attacked are named instances of regrettable community malfunctions stripped of anything personal to an individual that hurt both current users of Nix and people who might experiment with systems they can reason about, saw an interaction like this, and did something less painful with their day. Because my evident and sincere concern for the future of Nix is framed as stark disagreement over a few particularly sacred cows.
At least on `comp.lang.lisp` you got your middle finger alongside a worthwhile education:
"There are some things in life that you do not do if you want to be a moral being and feel proud of what you have accomplished."
My reaction was negative when I read the title (policy!), but the actual outcome was very reasonable. "Hey, can we add $ORIGIN to the VFS layer to support relocatable interpreters?" "You can already do that with binfmt_misc and ebpf, here's an example."
Then, later, a couple of patches to make binfmt_misc more ergonomic. Seems like a good outcome.
Nice, PT_INTERP is the only non-relocatable thing of ELF files and typically requires wrapper scripts/executables.
Regarding shebangs, I've never understood why the kernel cannot resolve e.g. `#!sh` relative to PATH instead of CWD. Posix prescribes that you should look for `sh` in PATH and don't expect it to be in `/bin/sh`. And using `/usr/bin/env sh` has the same issue: what if coreutils is installed elsewhere.
I suspect it's a mix of historical reasons (#! support was added pretty early, in 1980 or so, when the unix development philosophy I think tended quite strongly to "do the simple thing", and there wasn't so much variation in where you might put important binaries like the shell), plus the fact that the kernel doesn't know anything about PATH (it's only your shell that does), plus vague worries about potentially accidentally breaking existing #! lines that used to work.
> plus the fact that the kernel doesn't know anything about PATH (it's only your shell that does)
glibc is what deals with PATH, not the shell. On Linux, it's execvp(3) in libc which is implemented in terms of execve(2) in the kernel, but POSIX doesn't make a distinction between syscalls and library functions, so a kernel could implement either or none directly.
A surprising number of things in Linux are transparent to the kernel as they are implemented in user space. For instance, the Linux kernel doesn't support multithreading. It only supports processes sharing an address space. Which is what most people call threads - but in the kernel, they are just processes that happen to share an address space.
> the Linux kernel doesn't support multithreading. It only supports processes sharing an address space. Which is what most people call threads - but in the kernel, they are just processes that happen to share an address space.
This isn’t really true. It is just that internally, the kernel calls the thread ID “pid” and the process ID “tgid” (“thread group ID”). But the getpid system call returns the “tgid” not the “pid”, and to get the “pid” you need to call gettid (get thread ID). Other kernel interfaces also use the terminology “thread”, e.g. /proc/thread_self, set_tid_address, set_thread_area/get_thread_area, the CLONE_THREAD flag to clone, inter alia
Many people here seem to think this change will allow for people to package their Linux apps to be portable across distros/distro releases. If this is true then this a great!
On Debian-based distros it kind of sucks when the version of a package is only available in a newer release so the only way to get it is either compile it yourself or upgrade your system.
I remember trying to hack around this once by downloading dependencies from a newer release and helping the program locate them but I think I ran into a issue with the linker version being hardcoded or something and gave up.
>Many people here seem to think this change will allow for people to package their Linux apps to be portable across distros/distro releases.
There is no need to make binaries portable across the distros, because Linux is a source based OS. You can always just compile anything for your own system, just don't forget to submit the buildscript to the repo.
I wonder if something equivalent exists for loading other kinds of assets. Take files that would normally live under `/usr/share`. References to these files are typically hardcoded with an absolute path as well. It's usually possible to relocate them at build time by specifying a different absolute prefix, but making them relative the conventional way would likely require patching.
Sure, assuming this is your own software. The context of the article is on working towards a general approach to relocatable nix binaries so I don't think that's going to scale across nixpkgs.
Its unfortunate that this is using eBPF and not cBPF. I'd want to enable this in a namespace using binfmt-misc's namespace support, but eBPF requires root.
This is really awesome, can't wait for this to land in the linux kernel.
I have made appseed[0] (asciinema link[1]) which used zapps[2] which did a lot of tricks to achieve in end what this kernel patch could help in doing and hopefully more in the future!
My appseed project didn't work for large projects like OBS but I feel like the approach that this does could lead to mass adoption given that its now a kernel feature. It could in theory allow more portable binaries across Linux which is really cool :)
linux ELF code is loading the ELF loader only. $ORIGIN may not be a good idea
since that would add more ELF complexity to the kernel.
If we are honest with ourself, ELF is the core of the issue: for executables and
dynamic libraries we _now_ know it is severely obsolete on modern hardware
architectures.
I am currently using my own format, excrutiatingly simple, no loader, basically a
program segment, "userland syscalls" with hardware CPU synchronization. I do wrap
the executables into ELF capsules to run them transparently. So simple a small RFC will be enough.
With that, I discovered that the hard part is c++ and other similar languages which are very expensive in runtime infrastructure and linking complexity. I
would need to build a mesa vulkan driver with that format, and it seems the blocker is c++ (and similar language namely with grotesque and absurd syntax
complexity). Thx to valve to have removed a lot of c++, for less c++... would
have been much better if plain and simple C.
I would love to learn more about this;
I'm deeply interested in replacing ELF with something either simpler or maybe leveraging sqlite for much of the functionality.
Once I have something satisfying, I'll post the "RFC-like" draft and preliminary reference toolchain on HN (well, if I can since HN is going 'whatwg cartel' web engines only then I may be blocked for good). For the moment, I am 'testing' all that while writting my own wayland compositor for linux (that's why I will try to build mesa AMD vulkan driver for this format... or stick to wl_shm in the end).
Well, for a new executable and dynamic library format, the computer language syntax complexity and its runtime complexity do matter A LOT, c++ is massive pain, and since, if I am not mistaken, sqlite is going microsoft rust, that's bad omens since microsoft rust syntax seems to be now as brain damaged than c++ syntax... but with an even worse runtime (unless all that is not actually true, I have not checked). That said, ISO f*cked up C with things like '__thread' which requires some level of OS support.
Nix has caused so much busywork to be created through the entire Linux ecosystem. Rather than fixing their own project they force work upon so many others.
Instead of looking for ways Nix could work with ensuring else they force everything else to be modified to work with Nix. You will constantly seem them trying to get projects to adapt a flake.nix or to replace bash scripts with a different shebang since they hardcode a single binary that is rare for people to use in the real world.
>Rather than fixing their own project they force work upon so many others.
Nix has put in a ton of work to make their project work, and the assumptions in the rest of the ecosystem have been fighting every step of the way. And I would argue they've been changing it for the better, considering that if you were to just 'try to make nix work' you'd probably mainly come to the conclusion that it was impossible to make work sensibly due to the huge number of places where hardcoded relationships have been baked in.
Many of those assumptions are standards such as the filesystem hierarchy (granted, that's an xkcd 927 situation), nix is a bit of a special snowflake that breaks things in subtle ways (e.g. [0][1]) and needs workarounds in various places.
Those workarounds are not necessarily onerous, but they would be unnecessary if it didn't deviate.
>and the assumptions in the rest of the ecosystem have been fighting every step of the way
There is an arrogance in the way of thinking that when you see everyone breaking your arbitrary rules you think to yourself "there is no way my rules are at fault, it's everyone else's fault for breaking them."
>due to the huge number of places where hardcoded relationships have been baked in.
And why are hardcoded relationships bad for developers or users of the operating system? "Because they break Nix" is not a good excuse. Plenty of other operating systems work just fine with such things hardcoded, I don't see developers complaining about these hardcoded things, and moving things around can only make things more confusing for the user.
To me it looks like Nix picked a poor design and instead of going back to the drawing board and trying to come up with a better design they have gone all in on trying to force their bad design to work.
How much work do you think iOS or Android spend on app packaging / installation per year. Despite having millions of more developers and billions of more users they spend orders of magnitude less than NixOS does.
> There is an arrogance in the way of thinking that when you see everyone breaking your arbitrary rules you think to yourself "there is no way my rules are at fault, it's everyone else's fault for breaking them."
It's not really arrogance - it's just a different way to design a system. NixOS wants a few things:
1. An input-addressed dependency system
2. Easy and robust rollback/upgrades
3. Different versions of software/libraries/services coexisting on a single machine.
A lot of this is just impossible to have together with the way software were written, so NixOS made the changes necessary for them. When it became obvious it might be useful to other people, they upstreamed it.
I do agree with you that the whole `#!/usr/bin/env bash` shebang shenanigan is kinda ass (my personal nixos machine has a `/bin/bash` alias because there's not enough time in my life to care about this issue). But I also think that NixOS is right here, shebangs requiring a full path is just a terrible design in a multi-user world. If different users want to have a different bash implementation, they should be able to!
---
Fundamentally, I think the NixOS OS design is actually beautiful. It's taking most of what makes modern phone OS so reliable (read-only system, A/B partitions, etc...), and bringing their ideas in a shape that is coherent with desktop/server OS design.
Can you precisely describe what the goal of #1 is. For flatpaks an app's dependency can be specified with a sha256 of the artifact or a specific commit of the dependency.
The correct fix is to store the metadata outside the CAS, in for example, the loader, which can trivially delegate to another loader (does on my machine now).
bazel's wrong solution is $ORIGIN, Nix's wrong solution is "floating" CA / "realizations".
The technical debt / precedent argument doesn't apply: Nix never had this bug (for once), you the choices on this bug are 1. port the bazel bug to Nix 2. don't port the bug.
I predict... the bug is a shoe in, the bug lands eight days a week.
I was confused on what $ORIGIN means, so, macroexpanding the article a bit
https://fzakaria.com/2026/06/21/nix-needs-relocatable-binari...
> The loader in Linux however natively supports the variable $ORIGIN which translates to “the directory containing the executable.”
https://man7.org/linux/man-pages/man8/ld.so.8.html
But, if ld.so supports $ORIGIN already, why does the kernel needs to support it also? Or rather, why can't the kernel leverage ld.so and do this entirely on userspace?
$ORIGIN was only supported when the loader was looking for other dependencies. The loader itself (field PT_INTERP) is loaded by the kernel. So prior to this change, every program must hardcode the absolute path to ld.so. With support for an $ORIGIN-relative loader, each program could use its own copy of ld.so.
Each program having its own loader is an anti pattern. Such a requirement is overkill. You can have a single loader that supports everything on the system.
Not with NixOS. ld.so is tied to a version of glibc, in ways that can be subtly incompatible. And nixos can have multiple glibc version installed on a single machine.
Besides, it allows for upgrades/downgrades to be done in a way that's much less error-prone.
"ld.so is tied to a version of glibc" is such a horrible GNUism.
Isn't that kinda to be expected if you want to provide dynamic loading functionality (dlopen)?
Is the windows situation really all that different/better (with GetProcAddress in kernel32.dll)?
In theory, ld.so could provide a stable interface to its dynamic loading capabilities independently of glibc. Then glibc would not have to be updated in concert with ld.so. There's no inherent reason that glibc should be the library shipping a dynamic linker - we could easily be in a world where libld was developed independently from libc.
On Windows, things are less modular, so it's less of an issue. That said, there's also weird shenanigans when it comes to the CRT (which can be statically linked) vs ntdll (which provides the actual linker implementation), that can make certain niche features of the linker misbehave (delay loading in particular is weird).
In practice, ld.so is a piece of glibc. If you want to build ld.so from scratch, you have to clone glibc or get a tarball of glibc, and configure and build glibc.
You could make the argument that the glibc developers should split ld.so into its own subprojects. What for? That would just bring the extra responsibility of making sure that variations in glibc version work with variations in ld.so version for whatever reason.
What would happen in practice is that they would keep their version numbers in lock step, and systems integrators would use the same version. While the upstream glibc has to go through a dance of pretending that someone cares about their independence.
still, eventually there's a need to support on one platform different ld.so/glibc pairs (even if they are API/ABI compatible)
it seems nixos could set up a wrapper that invokes the right ld.so based on the executable. though at this point they could probably edit the ELF binaries and patch the fixed path to ld.so when nix is installing the program.
yeah, it seems strange that this needs kernel support. but more eBFP extension points are usually welcome, so sure, why not?
> though at this point they could probably edit the ELF binaries and patch the fixed path to ld.so when nix is installing the program.
That's exactly what they're doing right now - though instead of being "when installing the program", it's "when compiling the program". The problem with hardcoding the path is that it pins the "nix store" (where nix installs all of its programs) to a hardcoded location. If you want to move it, you have to rebuild all your packages - which is suboptimal.
In theory, nix could have a system in place to just patch all binaries when moving the nix store, but that would be incompatible with content-addressed derivation and otherwise break some other nice properties of the nix store.
(author) You did a great job articulating the points!
Not my area, but isn't it really only because glibc doesn't maintain stable interfaces across versions? If it did, you absolutely could use the same ld.so with different glibc versions. But it doesn't, so here we are.
The C standard isn’t stable across versions.
extern int errno;
The Windows situation is way different: every process is supposed to link against kernel32.dll, that's the public interface to the kernel. In Linux, glibc is just one of many C stdlib implementations, you can have many versions of glibc on the same system, etc.
Sure, but if you want dynamic loading from your c stdlib (which is defensible IMO), and you want the behavior/implementation to match with the loader, then you need some kind of coupling somewhere no?
You could have a very slim libdlopen that is used by both loader and libc, but I don't really see how that's any improvement/much different.
That's how Windows handles it - the OS libraries are language runtime agnostic and do not export language specific symbols nor need them, and language runtimes are distributed often separately.
So your C stdlib does not have dlopen() (in fact, it's not really part of C stdlib in POSIX anyway!) but you can link with standard OS-level ABI to OS-provided runtime services like "find a library and symbol from it" - AmigaOS did something similar.
ld.so is a component of glibc!
>in ways that can be subtly incompatible
There is much more value to be had in making glibc properly backward compatible so you can have a single one that can be used with everything than trying to make it so that you can swap everything around, creating extra complexity and compatibility risks.
> […] in making glibc properly backward compatible […]
You could only do that going forward though, and would be stuck, at least for a time, with the historic versions that still need extra handling. In an ideal world that wouldn't be needed, but in an ideal world you'd not need multiple versions of glibc at all so we aren't there.
Support in the kernel means that it will work even if applied to old packages that for some other reason need a held-back version of glibc. As mentioned elsewhere it also gives the feature to #! directives in scripts too.
How is that more valuable? It comes with two big pitfalls:
- It would still not allow downgrades to work properly
- It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together
Meanwhile, having different ld.so has many good use-cases, like simplifying development of ld.so itself, allowing them to swapped during updates in ways that are safer, etc...
And the eBPF binfmt support is a rather simple, generic mechanism that is likely to have many other use-cases beyond ld.so. So it's overall a pretty good resolution to the issue?
>It would still not allow downgrades to work properly
If you really need this, then there are options like swapping the linker with an older version or making a hardcoded linker that now points to an older one.
>It would cause glibc/ld.so to have a harder time adding new features, as they now need to worry about incompatible versions being used together
Every language runtime has to care about not breaking compatibility with apps that have been released already. This is not a new or unique problem.
Unfortunately, Glibc and loader are linked together intrinsically in the Linux ecosystem. So, if you want to be able to launch a program reliably, shipping your own loader and libc might actually be the only way.
Glibc is backward compatible though, they even have symbol versioning to provide multiple versions of the same symbol. So as long as you have the same or a newer version of glibc (than what was built against) you should be good to go. And I don't remember hearing about breakages for this.
Other libraries on the system is far more hit and miss, but glibc is quite compatible.
The other way around is harder though, you can't take a program built against a newer glibc and run it on an older version. So you generally need to spin up a container with some LTS distro and build your binary in it if you want it to be maximally compatible. (However, zig apparently is able to deal with this by shipping a mapping between glibc versions and symbol versions and doing the linking themselves. You can even use zig to link rust code using cargo-zigbuild and get that benefit.)
But if you want to be maximally portable: static linking against musl. Though beware that many things are slower in musl, such as the allocator.
Glibc is the opposite of backwards compatible. This thread I was a part of explains some issues faced in the past: https://news.ycombinator.com/item?id=47029789
Reading the linked bug report about executable stacks they fixed it? So they did the right thing. I'm not saying there will never be bugs (no non-trivial software is bug free), but as long as those are handled correctly that seems reasonable to me.
No, glibc isn't backwards compatible; I've had instances when the loader would refuse to load the executable because the installed glibc is too new for it.
That's not supposed to happen. I would like to have more info.
If I had to choose between 16 dockers, or 16 processes with different ld.so, guess what I'm choosing ...
One point mentioned in the first link you included is support of $ORIGIN in #! scripts - that would need kernel support.
This also can be handled by binfmt_misc; I include a demo of it in a gist.
> why can't the kernel leverage ld.so and do this entirely on userspace?
Because Linux is completely independent from its user space. There is no "the loader in Linux", it just happens that GNU wrote one and it's widely used.
You have the right idea but the wrong specifics. The boundary you're alluding to isn't the kernel/userspace boundary, it's the `libc` boundary, which is admittedly privileged by convention if not by Ring 0.
On most Linux systems this is regrettably `glibc` (in a container where you get to choose everyone chooses the superior option of `musl`), on all Darwin systems this is `libstandard`.
This thread is more concerned with Linux, so you are probably dealing with `/lib64/ld-linux-x86-64.so.2`, which operates in userspace but is by convention and opacity quite clearly part of "the system".
The kernel modification is a much bigger, much weirder side effect of a weird Nix loyalty test around shebang lines in shell scripts, on which is has opted to be intentionally and violently incompatible with everything for no benefit other than incompatibility.
That’s a weird rant which is manifestly false. On Nix the benefit you get from the incompatibility is the system can roll forward and back while it’s running so you can do big changes (eg updating or rolling back) the entire system without affecting running processes. You can also run groups of processes in isolation with different sets (or versions) of packages from the rest of the system.
It’s fair to argue about whether those things are really important or whether nix does this in the best way, but to claim that the only benefit is incompatibility is just obvious nonsense.
The privileged path is already there. `/usr/bin/env` is no different from `/bin/bash`, `/bin/sh`, or any other ELF artifact at a known place. The argument is made, the argument is spurious. It is made in the other direction regarding the driver run path, just as religious, opposite ruling from purity court. No one even knows why, the trail goes cold in a mysterious 2012 commit about Mesa, it's literally a performance to bring the plane cargo back. Receipts for claim: #141803.
Disagreeing with you doesn't make something a rant. If it's not clear how deep my Nix expertise is I will demonstrate it to any level you like: I'm just as entitled to an opinion as you are and I would appreciate it if the nixpkgs community was a little less rude to anyone who disagrees with some dogma no matter their knowledge. The nixpkgs community is not regarded as healthy or friendly after an internecine faction war that split three ways twice, producing four implementations none of which work, and if that reputation is ever going to mend, it will be because less than every Nix person has precisely zero chill the nanosecond anyone disagrees with them about anything.
It's no different other than it's a layer of indirection which allows bash to change in a way that would not be possible if scripts directly referenced /bin/bash or /usr/bin/bash. The only thing you can't change is the /usr/bin/env binary itself but that's not something that changes very much if at all. On my nix box, bash is at /run/current-system/sw/bin/bash and "current-system" is a symlink to a particular build in /nix/store. So /usr/bin/env is creating a layer of indirection so if I rebuild the system, the previous version can be kept around and I can roll back instantly at the boot menu and it will roll everything back to a consistent package set. Now as I say, you could achieve this in other ways eg fs snapshots etc. But to say that the change doesn't give you anything is just very obviously wrong.
And you don't need to worry- it's perfectly obvious how deep your Nix expertise is.
You've just used a personal attack to shout down someone who disagrees in good faith over a distinction you simultaneously describe in your own words as "something you could achieve other ways" (yeah, more than a little bit) referencing an artifact that "is a symlink", "on my nix box".
The only thing I attacked are named instances of regrettable community malfunctions stripped of anything personal to an individual that hurt both current users of Nix and people who might experiment with systems they can reason about, saw an interaction like this, and did something less painful with their day. Because my evident and sincere concern for the future of Nix is framed as stark disagreement over a few particularly sacred cows.
At least on `comp.lang.lisp` you got your middle finger alongside a worthwhile education:
"There are some things in life that you do not do if you want to be a moral being and feel proud of what you have accomplished."
- Erik Naggum
My reaction was negative when I read the title (policy!), but the actual outcome was very reasonable. "Hey, can we add $ORIGIN to the VFS layer to support relocatable interpreters?" "You can already do that with binfmt_misc and ebpf, here's an example."
Then, later, a couple of patches to make binfmt_misc more ergonomic. Seems like a good outcome.
Nice, PT_INTERP is the only non-relocatable thing of ELF files and typically requires wrapper scripts/executables.
Regarding shebangs, I've never understood why the kernel cannot resolve e.g. `#!sh` relative to PATH instead of CWD. Posix prescribes that you should look for `sh` in PATH and don't expect it to be in `/bin/sh`. And using `/usr/bin/env sh` has the same issue: what if coreutils is installed elsewhere.
I suspect it's a mix of historical reasons (#! support was added pretty early, in 1980 or so, when the unix development philosophy I think tended quite strongly to "do the simple thing", and there wasn't so much variation in where you might put important binaries like the shell), plus the fact that the kernel doesn't know anything about PATH (it's only your shell that does), plus vague worries about potentially accidentally breaking existing #! lines that used to work.
> plus the fact that the kernel doesn't know anything about PATH (it's only your shell that does)
glibc is what deals with PATH, not the shell. On Linux, it's execvp(3) in libc which is implemented in terms of execve(2) in the kernel, but POSIX doesn't make a distinction between syscalls and library functions, so a kernel could implement either or none directly.
To be fair, many shells have command caching, so they effectively roll their own exec syscall wrapper:
The kernel never looks at PATH, that's why. It's a shell construct.
... and the question was: "Why doesn't the kernel look at PATH"
It's not a "shell construct" either, the standard execvp libc function looks at PATH.
A surprising number of things in Linux are transparent to the kernel as they are implemented in user space. For instance, the Linux kernel doesn't support multithreading. It only supports processes sharing an address space. Which is what most people call threads - but in the kernel, they are just processes that happen to share an address space.
> the Linux kernel doesn't support multithreading. It only supports processes sharing an address space. Which is what most people call threads - but in the kernel, they are just processes that happen to share an address space.
This isn’t really true. It is just that internally, the kernel calls the thread ID “pid” and the process ID “tgid” (“thread group ID”). But the getpid system call returns the “tgid” not the “pid”, and to get the “pid” you need to call gettid (get thread ID). Other kernel interfaces also use the terminology “thread”, e.g. /proc/thread_self, set_tid_address, set_thread_area/get_thread_area, the CLONE_THREAD flag to clone, inter alia
Indeed, it's a libc construct.
Many people here seem to think this change will allow for people to package their Linux apps to be portable across distros/distro releases. If this is true then this a great!
On Debian-based distros it kind of sucks when the version of a package is only available in a newer release so the only way to get it is either compile it yourself or upgrade your system.
I remember trying to hack around this once by downloading dependencies from a newer release and helping the program locate them but I think I ran into a issue with the linker version being hardcoded or something and gave up.
>Many people here seem to think this change will allow for people to package their Linux apps to be portable across distros/distro releases.
There is no need to make binaries portable across the distros, because Linux is a source based OS. You can always just compile anything for your own system, just don't forget to submit the buildscript to the repo.
Wait, does this mean we can „virtualize“ ld.so now? This could finally solve the compatibility issues glibc causes.
(author) YESSSSSSS
it's finally the year of linux on the desktop! :)
I wonder if something equivalent exists for loading other kinds of assets. Take files that would normally live under `/usr/share`. References to these files are typically hardcoded with an absolute path as well. It's usually possible to relocate them at build time by specifying a different absolute prefix, but making them relative the conventional way would likely require patching.
You can always set "RESOURCE_PATH" and load your files from there.
Sure, assuming this is your own software. The context of the article is on working towards a general approach to relocatable nix binaries so I don't think that's going to scale across nixpkgs.
If you don't own the software, the software can do many things to break nix' relocatable binaries too.
Its unfortunate that this is using eBPF and not cBPF. I'd want to enable this in a namespace using binfmt-misc's namespace support, but eBPF requires root.
This is really awesome, can't wait for this to land in the linux kernel.
I have made appseed[0] (asciinema link[1]) which used zapps[2] which did a lot of tricks to achieve in end what this kernel patch could help in doing and hopefully more in the future!
My appseed project didn't work for large projects like OBS but I feel like the approach that this does could lead to mass adoption given that its now a kernel feature. It could in theory allow more portable binaries across Linux which is really cool :)
[0]: https://github.com/SerJaimeLannister/appseed
[1]: https://asciinema.org/a/zK0T3WXAxalMYCVUY1vp8FAv3
[2]: https://zapps.app/
linux ELF code is loading the ELF loader only. $ORIGIN may not be a good idea since that would add more ELF complexity to the kernel.
If we are honest with ourself, ELF is the core of the issue: for executables and dynamic libraries we _now_ know it is severely obsolete on modern hardware architectures.
I am currently using my own format, excrutiatingly simple, no loader, basically a program segment, "userland syscalls" with hardware CPU synchronization. I do wrap the executables into ELF capsules to run them transparently. So simple a small RFC will be enough.
With that, I discovered that the hard part is c++ and other similar languages which are very expensive in runtime infrastructure and linking complexity. I would need to build a mesa vulkan driver with that format, and it seems the blocker is c++ (and similar language namely with grotesque and absurd syntax complexity). Thx to valve to have removed a lot of c++, for less c++... would have been much better if plain and simple C.
I would love to learn more about this; I'm deeply interested in replacing ELF with something either simpler or maybe leveraging sqlite for much of the functionality.
Feel free to email me! (author)
Once I have something satisfying, I'll post the "RFC-like" draft and preliminary reference toolchain on HN (well, if I can since HN is going 'whatwg cartel' web engines only then I may be blocked for good). For the moment, I am 'testing' all that while writting my own wayland compositor for linux (that's why I will try to build mesa AMD vulkan driver for this format... or stick to wl_shm in the end).
Well, for a new executable and dynamic library format, the computer language syntax complexity and its runtime complexity do matter A LOT, c++ is massive pain, and since, if I am not mistaken, sqlite is going microsoft rust, that's bad omens since microsoft rust syntax seems to be now as brain damaged than c++ syntax... but with an even worse runtime (unless all that is not actually true, I have not checked). That said, ISO f*cked up C with things like '__thread' which requires some level of OS support.
Nix has caused so much busywork to be created through the entire Linux ecosystem. Rather than fixing their own project they force work upon so many others.
Instead of looking for ways Nix could work with ensuring else they force everything else to be modified to work with Nix. You will constantly seem them trying to get projects to adapt a flake.nix or to replace bash scripts with a different shebang since they hardcode a single binary that is rare for people to use in the real world.
>Rather than fixing their own project they force work upon so many others.
Nix has put in a ton of work to make their project work, and the assumptions in the rest of the ecosystem have been fighting every step of the way. And I would argue they've been changing it for the better, considering that if you were to just 'try to make nix work' you'd probably mainly come to the conclusion that it was impossible to make work sensibly due to the huge number of places where hardcoded relationships have been baked in.
Many of those assumptions are standards such as the filesystem hierarchy (granted, that's an xkcd 927 situation), nix is a bit of a special snowflake that breaks things in subtle ways (e.g. [0][1]) and needs workarounds in various places. Those workarounds are not necessarily onerous, but they would be unnecessary if it didn't deviate.
[0] https://github.com/NixOS/nixpkgs/issues/420513 [1] https://github.com/NixOS/nixpkgs/issues/94228
>and the assumptions in the rest of the ecosystem have been fighting every step of the way
There is an arrogance in the way of thinking that when you see everyone breaking your arbitrary rules you think to yourself "there is no way my rules are at fault, it's everyone else's fault for breaking them."
>due to the huge number of places where hardcoded relationships have been baked in.
And why are hardcoded relationships bad for developers or users of the operating system? "Because they break Nix" is not a good excuse. Plenty of other operating systems work just fine with such things hardcoded, I don't see developers complaining about these hardcoded things, and moving things around can only make things more confusing for the user.
To me it looks like Nix picked a poor design and instead of going back to the drawing board and trying to come up with a better design they have gone all in on trying to force their bad design to work.
How much work do you think iOS or Android spend on app packaging / installation per year. Despite having millions of more developers and billions of more users they spend orders of magnitude less than NixOS does.
Edit: I can't reply since I'm rate limited.
Debian already supports installing multiple php versions.
You can use the update-alternatives command to set which one is default for the php command.
Via your web server's config you can config which php version a site will use.
> There is an arrogance in the way of thinking that when you see everyone breaking your arbitrary rules you think to yourself "there is no way my rules are at fault, it's everyone else's fault for breaking them."
It's not really arrogance - it's just a different way to design a system. NixOS wants a few things:
1. An input-addressed dependency system
2. Easy and robust rollback/upgrades
3. Different versions of software/libraries/services coexisting on a single machine.
A lot of this is just impossible to have together with the way software were written, so NixOS made the changes necessary for them. When it became obvious it might be useful to other people, they upstreamed it.
I do agree with you that the whole `#!/usr/bin/env bash` shebang shenanigan is kinda ass (my personal nixos machine has a `/bin/bash` alias because there's not enough time in my life to care about this issue). But I also think that NixOS is right here, shebangs requiring a full path is just a terrible design in a multi-user world. If different users want to have a different bash implementation, they should be able to!
---
Fundamentally, I think the NixOS OS design is actually beautiful. It's taking most of what makes modern phone OS so reliable (read-only system, A/B partitions, etc...), and bringing their ideas in a shape that is coherent with desktop/server OS design.
I suggest you take a look at Fedora Silverblue. It accomplishes all 3 of those goals.
silverblue doesn't accomplish goal 3 either... if you need conflicting dependencies to interact in the same environment (i.e. outside a container)
I don't see how it achieves goal #1, what am I missing?
Can you precisely describe what the goal of #1 is. For flatpaks an app's dependency can be specified with a sha256 of the artifact or a specific commit of the dependency.
You kind of argue it’s okay one cannot eg. Install two different versions of PHP on a Debian system. Nay!
Of course you can.
Nobody forces projects to accept changes that add support for various platforms, yet it's common that they do.
Nix is one such platform.
The correct fix is to store the metadata outside the CAS, in for example, the loader, which can trivially delegate to another loader (does on my machine now).
bazel's wrong solution is $ORIGIN, Nix's wrong solution is "floating" CA / "realizations".
The technical debt / precedent argument doesn't apply: Nix never had this bug (for once), you the choices on this bug are 1. port the bazel bug to Nix 2. don't port the bug.
I predict... the bug is a shoe in, the bug lands eight days a week.