The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes?
(For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.)
> The nice thing about Python packaging is that it’s standardized
But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.
Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
> I don’t understand why Gentoo needs to rebuild every single release of uv-build.
Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
> you can do a normal version bound in your Python build backend metadata
Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".
distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to.
Gentoo builds every package from source on every user's machine, and when each user does an update, every package that changed gets rebuilt from source again.
So every Gentoo user that has a Python project that uses uv-build ends up having to rebuild uv-build, and then every Python project they have that uses it, every time uv-build pushes an update.
Other distros don't necessarily have quite the same problem because, if they include packages of Python projects, they don't normally track the build backend as a dependency. So if uv-build pushes an update, the distro would push an updated package for uv-build, but they wouldn't look at every Python project they package to see if it used uv-build as a build backend and rebuild it if it did. It's the "rebuild from source every time" aspect of Gentoo that forces that connection.
i guess gentoo is rolling-release, vs something like debian where every package is frozen in time (+ backports), and debian does seem to largely solve the coordination problem by making everything on the system use a fixed version of some package; with rare situations allowing >1 version of the package at a time.
my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.
This is what i mean when it seems typical for a distro to monkey around with packages and/or their build system. These were just the first two packages i clicked on in dev-python. maybe others are "pure upstream", but there's definitely the capability here to just... not have the problem?
> my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.
Thing is, that apparently won't work. From [0], with some more discussion starting at [1]
> > I don’t understand why Gentoo needs to rebuild every single release of uv-build.
> Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
gentoo also wrap python packages in their own package manager, packages in this tree appear to come from their upstream version control system eg Github, not PyPI.
just because someone on hn said something, doesn't make it true.
Per my response, that isn’t how Python packaging works. There is no “use the latest version” mechanism in standard Python packaging; leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.
> they don't normally track the build backend as a dependency... It's the "rebuild from source every time" aspect of Gentoo that forces that connection
I don't follow. This feels like Gentoo is mishandling build-tool dependencies.
Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
> This feels like Gentoo is mishandling build-tool dependencies.
I would say it's doing what it believes is necessary for the Python ecosystem. The build backend is an explicit part of the specification of a Python project.
> Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
No, because C/C++ libraries don't have an explicit specification of the build tool to be used in their project, the way a Python project does.
> The build backend is an explicit part of the specification of a Python project
Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update - that's a behaviour they've decided is correct on their own.
> C/C++ libraries don't have an explicit specification of the build tool to be used in their project
They don't have a standard way to specify such, but they have an implicit dependency on the installed compiler toolchain, and it would be just as valid to interpret "no way to specify a build-system dependency" as "always rebuild" when the build tool updates
I don't either. I used to run Gentoo for quite a few years in the past and build dependency updates would not trigger rebuilds of build-dependent packages.
Rebuilding the entire system with a new GCC was recommended by the documentation, but there was nothing in Portage forcing it. Neither would you rebuild the system if, e.g. make got an update.
No, and not with clang or llvm or rust. When we get a new Golang tho, that does rebuild stuff built with Go. And big perl updates cause a rebuild for perl packages.
IIRC, rebuilds to GCC outputs are/were only recommended when the compiler "profile" changes. So basically whenever the ABI could no longer be guaranteed.
At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.
(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)
> it seems extremely unlikely that they actually need to build every single patch version of uv.
If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.
This response downplays the portability concern which is the main point of the article. The ask for Gentoo to change its release & update cadence doesn't answer his plea.
As I see it, uv-build is Rust and will stay Rust, so on SPARC or address-space-constrained 32-bit targets may never work right. Astral prioritizes its users who run on tier-1 targets, and the cost of their uv-init default lands on the downstream packages that Gentoo has to build. Gentoo wants projects to choose pure python backends to reduce that ecosystem load, Astral would prefer if more projects default to uv-build (IIRC the default went from hatchling to uv-build in mid-2025).
Long-tail & evolution of ecosystem would be a far more interesting discussion, but the broader community has a "we are the future/adapt to us" sentiment that forecloses trade-offs.
I mean, SPARC and anything else can work right. But what uv can’t do is conjure up support for platforms that the larger ecosystem won’t even claim to support.
To that extent, I think uv (and CPython, and Rust, and so many other projects) are essentially trying to be honest in saying “we have no way to build and test that to our own standards.” I think that kind of honesty in validation is its own kind of load reduction on the ecosystem, because it means less tacit assumption that the maintainers will support anything that happens to build.
The problem is packages that use the build backends, and Gentoo is generally not in the business of maintaining that kind of patch.
Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps...
I think that’s actually a different problem: packages that use uv-build for their backend should use a semver range, which enables the kind of “pick a version and use it” strategy I mentioned before.
(This is the default when you initialize a project with uv, so I think someone would need to go out of their way to configure their build backend to request a specific patch version of uv-build.)
> packages that use uv-build for their backend should use a semver range
This wouldn't necessarily solve Gentoo's problem. The only way to be sure to solve Gentoo's problem would be for every Python project that uses uv-build to specify one particular version. Then Gentoo could be sure that once that release of that project was built once, it would never change.
In other words, for a distro like Gentoo, the build backend is a dependency, and if it changes, that triggers a change in the build, even if the build backend change would not actually change the final wheel that got built.
That would appear to be the polar opposite of effective: that turns a global constraint problem (everybody needing to have or install a compatible build backend version) into a global coordination problem (everybody needing to have or install an exact version). I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked.
It seems much easier to solve the problem of compatibility than the one of getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.
> I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked.
This doesn't make sense. Gentoo doesn't "pick" any build backend version. They build each Python project exactly the way the project owner tells them to, using whatever build backend is specified in pyproject.toml for that project. That's why it's such a burden for them (or more precisely their users, since each Gentoo user is having to do this) if build backend versions change so often and projects don't pin a specific version but just say "use the latest"--because then they have to use what the project owner says, and the project owner says, "Update to a new build backend every week because that's how often my chosen backend, uv-build, likes to push updates".
> if build backend versions change so often and projects don't pin a specific version but just say "use the latest"
This isn’t what’s happening: uv defaults to semver bounds, so a maintainer would have to go out of their way to introduce a troublesome backend constraint.
And note: there is no “use the latest” constraint in Python packaging. If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.
(I also know from first-hand experience that Gentoo and other distributions do not build Python packages as-is; they patch them in all kinds of ways. But that’s neither here nor there.)
But a new patch release will be within the same semver bound, so it amounts to the same thing.
> If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.
> But a new patch release will be within the same semver bound, so it amounts to the same thing.
Sorry, can you lay out the actual problem you’re envisioning for me? If uv-build goes from 0.13.1 to 0.13.2, then anybody with a semver bound of `uv-build>=0.12,<0.14` doesn’t need to update anything. They can keep using whatever version of the build backend they already have that satisfies that bound.
The main thing that would be troublesome is if a user pins their build backend to a specific range or version, e.g. `uv-build==0.13.1`. Then, upon bumping it, everyone who has already built 0.13.1 will need to invalidate and rebuild their build backend. But that’s why that isn’t the default.
> What do build frontends actually do, though?
uv will definitely use a compatible version. It won’t even download a copy of that that compatible version by default, since the uv binary itself is typically compatible and can invoke the build backend directly.
I’m not sure what pip and others do, but I would expect they do the same. It’s the obvious way to implement backend resolution.
> It’s the obvious way to implement backend resolution.
If a compatible version is already present, yes, a frontend can just use it if it's within the semver range specified.
But in many cases, builds are done in a fresh virtual environment that has nothing installed. In such a case, the only option is to download the latest version that's in the semver range specified. That's how the "build" frontend works, for example, unless you specify a flag to keep it from creating a fresh virtual environment.
Is that what Gentoo is doing here? I thought the whole point of the distro doing any sort of Python packaging is that they can pierce abstractions like virtual environments where it suits them.
(But also: if we’re talking virtual environments then we’re firmly in Python land, not distro land, where the correct answer is “use a wheel” instead of building it yourself.)
The whole point of Gentoo is to build everything directly from source. That means doing whatever the build source tells you to do. It also means not using a pre-built wheel, since that's not source code.
From what I can gather, Gentoo's Python build system does use a virtualenv [1]. Maybe that's not your definition of "distro land", but it appears to be Gentoo's.
It's not clear to me exactly what frontend build tool is used, whether it's one of the well known ones or something that the Gentoo developers coded themselves. However, if it's using a virtualenv, it's going to have to re-fetch the build backend and rebuild it.
> a global coordination problem (everybody needing to have or install an exact version).
This coordination problem is already solved by PyPI. That's where Gentoo looks to find the build backend--just like everyone else that builds packages. The project owner had to be able to find whatever version they used there or they couldn't even build their project locally. So they should be able to depend on the fact that anyone else trying to build their project can also find that same version at the same place.
> getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.
You're missing my point. If I know that my project builds properly with build backend X version Y, and I pin that version in my pyproject.toml, I don't care how many newer versions you want to release. Unless I make a change to my project that merits updating my build backend to a newer version, I'm going to simply leave my project alone. I already know it will build fine with the version I have. Why should I push an update every Tuesday just because you decided to release a new version of the build backend, which I don't need anyway?
It looks like it downloads from uv's git repository, which would also solve the coordination problem under discussion, since once a version is posted there it should remain there.
I hear you, but realistically the rustification of Python package internals is only going to strongly increase over the next year (I'm working on a number of such packages, myself), so systematic solutions to such annoyances are going to become increasingly necessary
> The nice thing about Python packaging is that it’s standardized
And a problem with Python packaging is that it's far too flexible.
For a pure Python package, IMO every single file in the wheel should be bit-for-bit identical to a file in the source tree (with the possible exception of a manifest listing the files). If this were actually a guarantee, even just for a category of package that opted in, then:
a) The build "tool" could be replaced with a build file selector, which would often be completely trivial and could have default behavior that worked for almost every pure python package right in pyproject.toml and
b) Instead of the giant mess of sigstore and attestation and weird trusted builds we have today, we would have a straightforward proof that a wheel actually matched a git commit irrespective of who built it. In fact, the proof could be embedded in the wheel! And the proof could be verified without executing any package-controlled code.
You can demonstrate it somewhat empirically: pip, uv, etc. all understand the build backend interface and packaging standards and can install any package that conforms with the standards.
(I think so much of the confusion around Python packaging comes down the layer of abstraction/standardization not being the one that users intuitively expect. This is unfortunate and is made worse by lackluster user-facing documentation. But there are standards that guarantee interoperability, and they’re pretty close to Pareto-optimal in terms of making all involved parties happy.)
Yeah, I think conformance is pretty good these days. PyPI itself enforces metadata well-formedness, and projects that don’t conform tend to get noticed quickly (because they don’t install correctly).
My somewhat ignorant experience is full of dealing with combinatorial expansion of python build backends and needing to husband them in nixos including all sorts of overrides to deal with, iirc, weird circularities in deps.
Yes, and unlike NixOS they dont calculate build hashes for the sake of making binary caches. So truly, you have to build from source, even if your recipe is exactly the same as everyone else's. I've never really understood this zealotry around compiling things myself, and boy are there some big (slow to compile) programs out there these days.
Sure, but again, you could in theory distribute said builds and stop wasting compute compiling yourself. I know nixpkgs, for example, builds with very generic targets and not super aggressive compiler flags, but at least the machinery exists to do this.
> Sure, but again, you could in theory distribute said builds and stop wasting compute compiling yourself.
Here are three links that you will find informative. [0][1][2]
While I'm not using the official Gentoo binhost, I've had one running on my LAN for my own personal use for roughly a decade now. It has been possible to set up a binhost for quite a bit longer than that... that's just when I got multiple machines with the same CPU in them, so it made sense for me to set one up.
On my 10 computers for example? :-) Gentoo wasn't aimed at people with such luxuries.
I'm not a Gentoo user BTW but I do think building from source is attractive for the same reasons cargo (Isn't that build from source mostly?) and go do it.
I wasn't there when it started but wasn't the raison d'etre was the GCC fork that eventually became the new upstream for GCC. Because the fork was faster Gentoo got the reputation that compiling from source was to get faster executable. This is a misunderstanding imho, as the greatest strength of Gentoo the level of customization enabled by use flags.
Gentoo was the first distro to use egcs as part of their default toolchain. Not sure if it was the only, but it was the only one I heard to had done so.
Python build backends are standardized in PEP 517, and packaging metadata is standardized in PEP 621. You can read the actual metadata PEPs for “computed” metadata i.e. the metadata that appears in distributions as well, if you’d like.
(It’s not kind or charitable to call someone a liar.)
Each build backend can have its own extensions to the standard packaging metadata. In practice this is common because Python packaging wasn’t designed to solve language ecosystem problems, but to make the union of maintainers, Linux distros, labs, and everybody else who uses Python somewhat happy and able to preserve existing flows.
This is bad for obvious reasons. But it’s not the same thing as Python packaging not being standardized; I would say that Python packaging is as standardized as its user base is willing to stomach. That willingness has grown over the years, to the immense credit of a large group of users who otherwise don’t agree on much.
You pretend not to understand what "packaging" means. No, it's not just the format description, even though the format description incorporates useless fields that are designed specifically to help setuptools to overcome the problems they created for themselves (all the "Dynamic" stuff is worthless otherwise).
Packaging, in this context, means a unified interface that describes to the tool what needs to happen to the source code in order for it to be transformed into a binary package. There are about a dozen tools that can package Python binary modules, setuptools, SWIG, bind11 and so on. Each does it in a completely different way. In practice, if you want to build the packages now available on PyPI, you will not be able to come up with a single tool that can build every one of them, probably, not even half of them.
All this talk about standardized metadata is worthless because of what actually goes into the package. The metadata is the least of your problems when packaging Python code.
> In practice, if you want to build the packages now available on PyPI, you will not be able to come up with a single tool that can build every one of them, probably, not even half of them.
This is the entire point of build backends, and is no longer true/hasn’t been true for several years. Build backends are not perfect, but they are the reason you can run `uv add` or `pip install` and get a functional wheel from an sdist on PyPI regardless of whether the sdist contains Python, C++, Rust, or anything else.
(This is now the second time you’ve imputed dishonesty in a conversation where I’ve tried to be helpful. So this will be my last response.)
Nobody cares about PEP-517 or any PEP for that matter since PyPA decided they are above it. PyPA decides how things are done, PEPs are just "historical documents".
Nobody cares if package metadata is standardized. The tools aren't. The tools are the problem, not the package format.
Well, if the poster says something that is blatantly untrue, how would you call them? Liar describes exactly someone who intentionally tells something that isn't true.
By a search, Gentoo accounts for less than 1% of all developers. If uv-build is more conformant to people, probably volunteers (and if they're using it, it probably is), I don't think the Gentoo people have any right to want them not to use it because it makes their lives harder. They can request, and they do request nicely, but they also present the drawbacks for them as if they are absolute, and even claim that "its benefits don’t outweigh the drawbacks".
> they do request nicely, but they also present the drawbacks for them as if they are absolute, and even claim that "its benefits don’t outweigh the drawbacks".
for some people the benefits absolutely do outweigh the drawbacks. Maybe not for you....but on the whole I don't know - if the whole issue had been solved without being given attention would you have ended up caring which way it went?
I'm sure it accounts for a much larger proportion of people using exotic or legacy systems. In any case, I think most people are using uv-build either because it's a default for uv now or just because it's hip and new. If a project absolutely needed it then so be it, but if you knew it was going to make life difficult even just for one person and no benefit to you then you'd probably choose something else.
Looks like it's being built with fat LTO by default (and gentoo patched to thin LTO). Which is fine if you want to spend CI cycles once to ship executables to thousands of users.
If you compile from source and run the tool maybe once then they should consider going further and run with maybe opt-level=1, lto=off; halfway between release and debug. This is how rustc itself builds some internal buildtools that get recompiled frequently relative to how often they run.
UV build is distributed as a binary/executable, e.g. manylinux wheels on pypi: https://pypi.org/project/uv-build/#files. If a user or distro's conventions are to build things from source, this is a consequence they have implicitly accepted. For the vast majority of computer users and uses, installing binaries makes more sense.
It is still a problem where there are no binaries for platforms otherwise supported by the distro, and in fact supported by the package being built - because uv-build is the only part that does not run on given platform and can't be reliably built for it
I'm not sure that skipping SPARC is a bad thing. Who even has SPARC machines sitting around to validate builds? Even Fujitsu moved to ARM for their later cores.
i got a debian maintainer for old IBM mainframes asking me to fix bugs and run tests for that architecture. ¯\_(ツ)_/¯ these people exist, but i'm not going to support them. feel free to fork me!
> for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.
maybe it's time to embrace the enemy and go "clanker, this is uv-build source, rewrite it in portable C". rewrite it in C FTW!
btw, I know of a guy who using extensive amounts of tokens to make his fork of mrustc usable via "claude, make this rustc compile 99% of crates.io", I think this also can yield some results.
I don't think I agree with the author's assessment here:
> The other reason is performance and that argument makes even less sense. Even if it mattered, we’re realistically talking about a difference in the ballpark of 100 ms, for an action executed rarely; it is equally possible that any gain will be outweighed by the time needed to fetch the 1.5 MiB wheel for uv-build.
Modern CI architectures are building Docker stacks from scratch and using a readthrough cache of wheels. Downloading may still be the dominant factor, but the second-dominant factor does start to add up, especially when the downloading gets cached at various layers.
I think mgorny is aware of that; he’s been active on the relevant DPO threads.
(At the end of the day, this is exactly what support tiers are for. CPython, at least to my eyes, has been very transparent about not guaranteeing support for every platform that has a C compiler.)
I found this recently - couldn't compile it on HPUX. Trying to get rust on there would have been an absolute impossibility of course. I can get bash though, and lots of other GNU utilities - all written in C of course. Thank goodness some people care!
People who don't care about HPUX will never care about it but they would if e.g. we decided that only windows and Ubuntu were supported ... except for the windows and ubuntu python users who wouldn't care.
It's I'm-all-right Jack until it's your use case that gets bulleted.
Hi, one of the co-maintainers of GNU coreutils here. It is nice to hear things worked on HP-UX. I don't have access to it, and I don't think the other maintainers do either. See this commit [1].
[1] https://github.com/coreutils/gnulib/commit/c599eba7bed3baa30...
Yes we're using quite an old version 8.13 and yet it does the job for our build which maintains an EXTREMELY important bit of software in public service relating to healthcare.
It's on Itanium so obviously we have to move away from that to Linux but the existence of open source tools like coreutils gives us time.
No such luck with python. We have to postpone some changes as a result.
As I say, there's some kind of effect where changes that are only negative on the margins are cheered until one finds oneself on one of those margins.
I think there’s a categorical difference between HP-UX (which seems to no longer even be maintained by HP) and something like Linux on the top 5-6 most popular architectures.
(But with all that said: there’s no ideological dispute here. If there was a reliable way to build and test for HP-UX, I assume a lot more people would declare support for it. The problem comes down to maintainers being asked to support platforms they ultimately have no ability to support, except in the weakest sense of “it compiles.”)
When you think about GNU autotools they're all about being able to compile on platforms that the software has never been compiled on before - that's why there are all those feature tests to see what the OS and compiler support.
I've done this many times on things like AIX 4.3 and HP-UX and so on. Sometimes you can get through the compile without any problems at all and the tool often works enough for your usecase and that's a win.
Sometimes there are particular little problems, sometimes it's something serious like an actual missing C library call or whatever and then you have to do some work. With nicely designed tools that have fairly wide support to begin with this tends to be easier.
Yes, optionally apparently but I think we'll end up with forks of Python if the Rust enthusiasts push it any further. I think the option will be off for me.
I presume that those who don't want to have to build rust just to get python will stick the rust code into an LLM and get out an acceptable analog in C.
The point of converting things is here already - existing code is like a very precise prompt. I've used it for Perl-Python mainly and the results are far better than trying to write new code.
Maybe it's time to grow up and download binary builds like a normal person. I had my 'let's compile the linux kernel from source for 3.5% hypothetical performance gains' phase but you can't expect professional devs to cater to source-obsessed hobbyists
I doubt this oft-repeated stereotype was every particularly accurate. I'm not aware of any current Gentoo users on the distro specifically for performance reasons. Gentoo users are in it for control over their system and customization.
> In the early days, there was much promotion of Gentoo as being “faster”, because you could compile everything based on your particular processor, etc. And you could increase the optimization level for a boost. Their web site still touts this as a reason to use Gentoo.
> In reality, the performance is more or less the same as on any other distribution. The folks who stick to Gentoo tend not to care about performance as much. Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site.
A friend of mine got me on Gentoo in 2004. He spent all of his time tweaking compiler flags like -O3 -ffast-math -fomit-frame-pointer -funroll-loops and running into so many issues in the process. I think he only lasted about 9 months on Gentoo. To this day, he's the only person I've ever met that was there for performance.
Good timing! If memory serves, you missed the Big Libpng Upgrade that either got the concept of Package Slots into Portage or convinced everyone that they were a _really_ good feature to have and use where appropriate.
> Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site.
Based on what I can see after putting a minute or two into searching, that verbiage being referred to only appears on the About page. [0]
The distro's home page says
Welcome to Gentoo, a highly flexible, source-based Linux distribution.
and has used those words since at least January 2023 [1]. Its home page has spoken about flexibility and has not spoken about performance since the big redesign in April 2016... seven years prior to the publication of that guy's blog post. [2][3]
And -speaking as a Gentoo user since 2002- neither the user community nor the maintainer community hold the opinion that Ultimate Maxxximum Performance is the distro's goal. Trouble reports that come from "ricers" pretty much always get told "Set your compiler flags to something sane (try '-O2 -pipe' or '-O2 -pipe -march=native') and rebuild. If you can still repro the problem, file a new report because this one is getting closed INVALID.".
And -while the old Gentoo Wiki burned down so I can't prove it- [4] the advice about GCC optimization in the wiki has always cautioned against using aggressive compiler flags and warned that doing so will -among other things- not substantially improve performance. The advice about reasonable compiler flags has pretty consistently been "Use -O2, -pipe, -march=$THE_CORRECT_SETTING_FOR_YOUR_CPU, and maybe -fomit-frame-pointer"... though the "-fomit-frame-pointer" advice has always come with a warning that doing that may make debugging either extremely difficult or impossible, and the "-march" advice the warning that this may make it impossible to run those binaries on another computer.
Anyway. There's only so much you can do to erase an -allegedly- pervasive myth. The fact that some folks have the belief that Gentoo Linux is all about Maxxximum Performance doesn't change the fact that nearly all of the people who use it and -almost certainly- all of the people who maintain it strongly disagree with that claim. The distro is and always has been about flexibility, user choice, and having fun. And it's a nice bonus if building for -say- -march=athlon-xp instead of -march=i586 happens to get you a bit of extra performance here and there.
[3] That year's April 1 redesign was a bit cool, IMO.
[4] The oldest copy that I can easily find is from mid 2013 [5], so you'll just have to trust me that this has been the long-standing opinion of the overwhelming majority of users and maintainers. While you're there, take a minute or two to skim the prose and notice how it consistently warns against using aggressive compiler flags.
The vast majority of desktop Linux users run Ubuntu (initial motto: Linux for human beings). So one may argue the literal opposite of what you wrote, in that if Canonical didn't invest into making a Linux distro that's accessible to normal people, it wouldn't have the relevance it now does. On top of that, Linux has (or had) a number of clear advantages over Windows that dwarf the presumed upsides of compiling everything from source vs. downloading builds.
There's not much of an argument for adopting obscure niche practices for their own sake
The rather large giant collosal planet sized argument that you somehow missed despite that, is that every single thing that you consider good and normal today doesn't exist except for the fact that at one point it was the obscure niche thing that annoyed intolerant "normal" people.
You, like the other commenter, disregarded the core argument of the matter in favor of contrarianness for its own sake: are you trying to argue that building literally everything from source is cool and good because it's niche and weird, whereas Linux was niche and weird once but it's cool and good now? As I said, Linux had and still has clear advantages over Windows to the point that Microsoft felt threatened by it fairly soon (cue Ballmer's chair throwing, cancer remark, etc.) and tried to undermine it before giving up with Nadella. What, exactly, are the obvious and objective upsides of compiling everything from source? Many posters in the thread dispelled performance-maxxing as a meme so it can't be that. You are comparing apples to oranges here.
I feel like I'm trying to explain a shade of colour to a blind person.
building from source has uses which you believe are worthless and other people don't think are worthless.
Anyone who uses cargo or installs a module for golang is building from source. But that, probably, is considered useful. It's only people who build C code from source who are niche and wierd. ....
Linux was obscure and niche for a very long time before Ubuntu existed and Ubuntu only existed because someone bothered to use and develop Linux when it wasn't normal.
The Ubuntu users are only there because after they had told the rest of us how silly we were for using Linux they eventually had to use it themselves for work.
Now we're going to listen to that sort of stuff again without rolling our eyes? ....
SLS for me. haha. :-) But now at least you have the choice! It's great really. I can use Artix because I like being nearer the bleeding edge but even that is really a lot easier to use than Linux was in the days when I had to compile my kernel.
Nevertheless I remember lots of my friends who started off saying Linux was hopeless and ended up using it and Ubuntu was their gateway. The Linux world now is full of people who are fairly conformist and don't care about how it all began and who don't value the idea that open source has margins from which who knows what world beating things might eventually come. The fact that margins can exist is what makes open source great.
Gentoo’s decision to support source builds seems eminently reasonable to me, and I don’t think we should reward it with condescension.
The bigger problem, to me, is that it’s not really clear why this is actually a problem for Gentoo. I am glad they perform user-facing source builds, however, because they see things that other distributors don’t.
Firefox built from source is sufficiently sped up that I can notice it starting up at least 1-2 seconds faster. Not that I use the distro for speed, but I'm not complaining about a faster browser.
If you insist on acting like a normal person, no one stops you from using something normal.
Gentoo made a lot of sense WAY back in a pre-docker world, where you had to run the whole world on the host OS. Being able to compile it all meant you could link it all together with whatever weird combination of optional libraries you wanted. Things like getting sendmail to auth over LDAP, that nobody packaged in the base distro, become possible. And with Gentoo's tooling, you could build a coherent distro to your own unique demands.
Now though? It's way easier to have a herd of unicorn docker containers running that each handle one weird thing in isolation.
The herd of docker containers would in practice share almost nothing apart from the kernel and waste a lot of RAM. Didn't matter when RAM was cheap, but nowadays...
Less and less with the ongoing ads, taskbar issues, and RAM eaating bloatware.
By just adding
preempt=full
to most GNU/Linux distros to the grub kernel boot options (trivial even for Fedora Silverblue, just run
sudo rpm-ostree kargs --editor
) and append at the end of these boot flags you are seeing the mentioned string. Done, multimedia ready desktop for tons of tasks outperforming Windows 11.
To copy my response on Lobsters:
(I am one of uv’s maintainers.)
The nice thing about Python packaging is that it’s standardized: if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
With that said, I don’t understand why Gentoo needs to rebuild every single release of uv-build. It doesn’t change as much as uv itself, and it’s intentionally compatible across uv versions. Why not set a policy of only updating once weekly, or even less frequently, with exceptions for important breaking changes or bug fixes?
(For context: the only reason uv-build releases as often as it does is because it’s easier to keep its internals in lockstep with the rest of uv. But you can do a normal version bound in your Python build backend metadata and uv will happily use whatever version of uv-build is already present.)
> The nice thing about Python packaging is that it’s standardized
But it isn't. Once upon a time, it was: you used distutils. But of course distutils had a lot of problems. Instead of adopting a single standardized solution for those problems, the Python developers punted and let third parties do it. So now we do not have a single standardized solution for building Python packages; we have a bunch of third party solutions, none of which is in the Python standard library.
Yes, all those third party solutions have to be PEP 517 compatible, but that's not standardization on a tool; it's just standardization on an interface between the backend and the frontend. Which is nice, but doesn't solve the problem under discussion here.
> if uv’s build backend doesn’t fit into the hole you want, then you can use any number of other backends that do an admirable job of conforming to the packaging standards.
As the project developer, sure. But as a Linux distro building your python package, Gentoo has to use the build backend you specify. It can't say, well, this particular package is just pure python, so really all we need is a minimal tool to build a pure python none-any wheel. It has to use the backend you specified.
> I don’t understand why Gentoo needs to rebuild every single release of uv-build.
Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
> you can do a normal version bound in your Python build backend metadata
Sure, but how many users of uv-build actually do that? Seems like the answer to that is "not enough to matter".
distros patch the build process when they package stuff for the distro all the time. why would they "have to" do anything? i'm not super familiar with gentoo but it seems to me like any other 90s era distro where the distro mediates the relationship between the user and the upstream developer for all packages, by applying patches and potentially doing whatever build they want to.
Gentoo builds every package from source on every user's machine, and when each user does an update, every package that changed gets rebuilt from source again.
So every Gentoo user that has a Python project that uses uv-build ends up having to rebuild uv-build, and then every Python project they have that uses it, every time uv-build pushes an update.
Other distros don't necessarily have quite the same problem because, if they include packages of Python projects, they don't normally track the build backend as a dependency. So if uv-build pushes an update, the distro would push an updated package for uv-build, but they wouldn't look at every Python project they package to see if it used uv-build as a build backend and rebuild it if it did. It's the "rebuild from source every time" aspect of Gentoo that forces that connection.
i guess gentoo is rolling-release, vs something like debian where every package is frozen in time (+ backports), and debian does seem to largely solve the coordination problem by making everything on the system use a fixed version of some package; with rare situations allowing >1 version of the package at a time.
my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.
for example, they seem to be fine doing a quick sed on pyproject.toml (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/pa...) or patching stuff for compat (https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/as...)
This is what i mean when it seems typical for a distro to monkey around with packages and/or their build system. These were just the first two packages i clicked on in dev-python. maybe others are "pure upstream", but there's definitely the capability here to just... not have the problem?
> my take on linux distros being annoyed by their own policies is that perhaps they should change the policy. in this case, perhaps they should pretend uv-build updates once a month or something.
Thing is, that apparently won't work. From [0], with some more discussion starting at [1]
> > I don’t understand why Gentoo needs to rebuild every single release of uv-build.
> Because if you don't pin a version of uv-build in your project, that means "use the latest version available". Gentoo can't second guess you--even if it were able to verify somehow that the new version of uv-build that just got released will end up producing an identical wheel for your project as the previous version, without actually using the new version of uv-build.
[0] <https://news.ycombinator.com/item?id=50077466>
[1] <https://news.ycombinator.com/item?id=50077553>
> Gentoo can't second guess you
gentoo appears to support exactly 7 versions of uv-build, a list they update by hand: https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...
git changelog of uv-build in the gentoo package manager: https://gitweb.gentoo.org/repo/gentoo.git/log/dev-python/uv-...
gentoo also wrap python packages in their own package manager, packages in this tree appear to come from their upstream version control system eg Github, not PyPI.
just because someone on hn said something, doesn't make it true.
Per my response, that isn’t how Python packaging works. There is no “use the latest version” mechanism in standard Python packaging; leaving a version unconstrained means that the installer, meaning Gentoo in this case, gets to decide how to constrain it.
> i guess gentoo is rolling-release
That's my understanding, yes.
> they don't normally track the build backend as a dependency... It's the "rebuild from source every time" aspect of Gentoo that forces that connection
I don't follow. This feels like Gentoo is mishandling build-tool dependencies.
Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
> This feels like Gentoo is mishandling build-tool dependencies.
I would say it's doing what it believes is necessary for the Python ecosystem. The build backend is an explicit part of the specification of a Python project.
> Does Gentoo also recompile every C/C++ library in the dependency tree every time GCC pushes a minor version bump?
No, because C/C++ libraries don't have an explicit specification of the build tool to be used in their project, the way a Python project does.
> The build backend is an explicit part of the specification of a Python project
Nothing in the Python spec forces Gentoo to invalidate the downstream package every time build-system dependencies update - that's a behaviour they've decided is correct on their own.
> C/C++ libraries don't have an explicit specification of the build tool to be used in their project
They don't have a standard way to specify such, but they have an implicit dependency on the installed compiler toolchain, and it would be just as valid to interpret "no way to specify a build-system dependency" as "always rebuild" when the build tool updates
> I don't follow.
I don't either. I used to run Gentoo for quite a few years in the past and build dependency updates would not trigger rebuilds of build-dependent packages.
Rebuilding the entire system with a new GCC was recommended by the documentation, but there was nothing in Portage forcing it. Neither would you rebuild the system if, e.g. make got an update.
No, and not with clang or llvm or rust. When we get a new Golang tho, that does rebuild stuff built with Go. And big perl updates cause a rebuild for perl packages.
IIRC, rebuilds to GCC outputs are/were only recommended when the compiler "profile" changes. So basically whenever the ABI could no longer be guaranteed.
Strictly speaking, they should, right? Differences in GCC could produce differences in anything it compiles.
Seems like a design flaw in gentoo that previously hadn't been an issue because build tools weren't being updated nearly as often as uv-build is.
I linked to some of the relevant standards below.
At the end of the day, it’s hard for me to even pinpoint what is problematic for Gentoo here: I can understand that they need to support multiple build backends as a reality of Python packaging and that’s frustrating, but per below it seems extremely unlikely that they actually need to build every single patch version of uv. Nothing about uv’s defaults or the packaging template would require that, and users would have to go out of their way to configure their build backend in that way.
(And note: this is not unique to uv! You can grief anybody who does an sdist build by pinning to a random ancient version of setuptools or any other build backend, including other ones that use native code. This is why uv, like everybody else, encourages semver ranges for build backends.)
> it seems extremely unlikely that they actually need to build every single patch version of uv.
If every new patch version is still within the semver range specified in a project that uses uv-build, then yes, every Gentoo user will have to rebuild that new patch version of uv-build in order to rebuild that project. And since the build backend is a dependency of the project, pushing a new patch version will trigger a rebuild of the project.
This response downplays the portability concern which is the main point of the article. The ask for Gentoo to change its release & update cadence doesn't answer his plea.
As I see it, uv-build is Rust and will stay Rust, so on SPARC or address-space-constrained 32-bit targets may never work right. Astral prioritizes its users who run on tier-1 targets, and the cost of their uv-init default lands on the downstream packages that Gentoo has to build. Gentoo wants projects to choose pure python backends to reduce that ecosystem load, Astral would prefer if more projects default to uv-build (IIRC the default went from hatchling to uv-build in mid-2025).
Long-tail & evolution of ecosystem would be a far more interesting discussion, but the broader community has a "we are the future/adapt to us" sentiment that forecloses trade-offs.
I mean, SPARC and anything else can work right. But what uv can’t do is conjure up support for platforms that the larger ecosystem won’t even claim to support.
To that extent, I think uv (and CPython, and Rust, and so many other projects) are essentially trying to be honest in saying “we have no way to build and test that to our own standards.” I think that kind of honesty in validation is its own kind of load reduction on the ecosystem, because it means less tacit assumption that the maintainers will support anything that happens to build.
The problem is packages that use the build backends, and Gentoo is generally not in the business of maintaining that kind of patch.
Admittedly rustification like that is a problem not only for gentoo, I have commercially deployed systems where one of the most annoyings elements of updating is presence of rust in python deps...
I think that’s actually a different problem: packages that use uv-build for their backend should use a semver range, which enables the kind of “pick a version and use it” strategy I mentioned before.
(This is the default when you initialize a project with uv, so I think someone would need to go out of their way to configure their build backend to request a specific patch version of uv-build.)
> packages that use uv-build for their backend should use a semver range
This wouldn't necessarily solve Gentoo's problem. The only way to be sure to solve Gentoo's problem would be for every Python project that uses uv-build to specify one particular version. Then Gentoo could be sure that once that release of that project was built once, it would never change.
In other words, for a distro like Gentoo, the build backend is a dependency, and if it changes, that triggers a change in the build, even if the build backend change would not actually change the final wheel that got built.
That would appear to be the polar opposite of effective: that turns a global constraint problem (everybody needing to have or install a compatible build backend version) into a global coordination problem (everybody needing to have or install an exact version). I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked.
It seems much easier to solve the problem of compatibility than the one of getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.
> I can imagine most of what causes problems for Gentoo is people trying to do the “right” thing of pinning an exact build backend version, which then invalidates whatever compatible build backend version they’ve picked.
This doesn't make sense. Gentoo doesn't "pick" any build backend version. They build each Python project exactly the way the project owner tells them to, using whatever build backend is specified in pyproject.toml for that project. That's why it's such a burden for them (or more precisely their users, since each Gentoo user is having to do this) if build backend versions change so often and projects don't pin a specific version but just say "use the latest"--because then they have to use what the project owner says, and the project owner says, "Update to a new build backend every week because that's how often my chosen backend, uv-build, likes to push updates".
> if build backend versions change so often and projects don't pin a specific version but just say "use the latest"
This isn’t what’s happening: uv defaults to semver bounds, so a maintainer would have to go out of their way to introduce a troublesome backend constraint.
And note: there is no “use the latest” constraint in Python packaging. If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.
(I also know from first-hand experience that Gentoo and other distributions do not build Python packages as-is; they patch them in all kinds of ways. But that’s neither here nor there.)
> uv defaults to semver bounds
But a new patch release will be within the same semver bound, so it amounts to the same thing.
> If you leave the build backend version unconstrained, the build frontend is allowed to resolve any version it sees fit, including the version it already has.
What do build frontends actually do, though?
> But a new patch release will be within the same semver bound, so it amounts to the same thing.
Sorry, can you lay out the actual problem you’re envisioning for me? If uv-build goes from 0.13.1 to 0.13.2, then anybody with a semver bound of `uv-build>=0.12,<0.14` doesn’t need to update anything. They can keep using whatever version of the build backend they already have that satisfies that bound.
The main thing that would be troublesome is if a user pins their build backend to a specific range or version, e.g. `uv-build==0.13.1`. Then, upon bumping it, everyone who has already built 0.13.1 will need to invalidate and rebuild their build backend. But that’s why that isn’t the default.
> What do build frontends actually do, though?
uv will definitely use a compatible version. It won’t even download a copy of that that compatible version by default, since the uv binary itself is typically compatible and can invoke the build backend directly.
I’m not sure what pip and others do, but I would expect they do the same. It’s the obvious way to implement backend resolution.
> It’s the obvious way to implement backend resolution.
If a compatible version is already present, yes, a frontend can just use it if it's within the semver range specified.
But in many cases, builds are done in a fresh virtual environment that has nothing installed. In such a case, the only option is to download the latest version that's in the semver range specified. That's how the "build" frontend works, for example, unless you specify a flag to keep it from creating a fresh virtual environment.
Is that what Gentoo is doing here? I thought the whole point of the distro doing any sort of Python packaging is that they can pierce abstractions like virtual environments where it suits them.
(But also: if we’re talking virtual environments then we’re firmly in Python land, not distro land, where the correct answer is “use a wheel” instead of building it yourself.)
The whole point of Gentoo is to build everything directly from source. That means doing whatever the build source tells you to do. It also means not using a pre-built wheel, since that's not source code.
From what I can gather, Gentoo's Python build system does use a virtualenv [1]. Maybe that's not your definition of "distro land", but it appears to be Gentoo's.
It's not clear to me exactly what frontend build tool is used, whether it's one of the well known ones or something that the Gentoo developers coded themselves. However, if it's using a virtualenv, it's going to have to re-fetch the build backend and rebuild it.
[1] https://projects.gentoo.org/python/guide/basic.html
> a global coordination problem (everybody needing to have or install an exact version).
This coordination problem is already solved by PyPI. That's where Gentoo looks to find the build backend--just like everyone else that builds packages. The project owner had to be able to find whatever version they used there or they couldn't even build their project locally. So they should be able to depend on the fact that anyone else trying to build their project can also find that same version at the same place.
> getting every open source maintainer to upgrade their build backend version (whether uv-build, or flit, or anything else) every Tuesday.
You're missing my point. If I know that my project builds properly with build backend X version Y, and I pin that version in my pyproject.toml, I don't care how many newer versions you want to release. Unless I make a change to my project that merits updating my build backend to a newer version, I'm going to simply leave my project alone. I already know it will build fine with the version I have. Why should I push an update every Tuesday just because you decided to release a new version of the build backend, which I don't need anyway?
gentoo does not appear to download uv from PyPi https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...
It looks like it downloads from uv's git repository, which would also solve the coordination problem under discussion, since once a version is posted there it should remain there.
I hear you, but realistically the rustification of Python package internals is only going to strongly increase over the next year (I'm working on a number of such packages, myself), so systematic solutions to such annoyances are going to become increasingly necessary
Well, it did force me to work on better build caching and the like, but at the same time, it accelerated my wish to remove python where possible
> The nice thing about Python packaging is that it’s standardized
And a problem with Python packaging is that it's far too flexible.
For a pure Python package, IMO every single file in the wheel should be bit-for-bit identical to a file in the source tree (with the possible exception of a manifest listing the files). If this were actually a guarantee, even just for a category of package that opted in, then:
a) The build "tool" could be replaced with a build file selector, which would often be completely trivial and could have default behavior that worked for almost every pure python package right in pyproject.toml and
b) Instead of the giant mess of sigstore and attestation and weird trusted builds we have today, we would have a straightforward proof that a wheel actually matched a git commit irrespective of who built it. In fact, the proof could be embedded in the wheel! And the proof could be verified without executing any package-controlled code.
> The nice thing about Python packaging is that it’s standardized
Has anybody done the analysis to determine if this is something that can, in practice, be relied on?
You can demonstrate it somewhat empirically: pip, uv, etc. all understand the build backend interface and packaging standards and can install any package that conforms with the standards.
(I think so much of the confusion around Python packaging comes down the layer of abstraction/standardization not being the one that users intuitively expect. This is unfortunate and is made worse by lackluster user-facing documentation. But there are standards that guarantee interoperability, and they’re pretty close to Pareto-optimal in terms of making all involved parties happy.)
> any package that conforms with the standards.
Right, but what I'm asking is: by and large, can you expect any random package on pypi to conform to the standard?
Yeah, I think conformance is pretty good these days. PyPI itself enforces metadata well-formedness, and projects that don’t conform tend to get noticed quickly (because they don’t install correctly).
My somewhat ignorant experience is full of dealing with combinatorial expansion of python build backends and needing to husband them in nixos including all sorts of overrides to deal with, iirc, weird circularities in deps.
Is Gentoo one of those compile-everything-from-source distros?
With apologies to Slackware, it is the compile-everything-from-source distro.
Happy user for over 23 years now!
Yes, and unlike NixOS they dont calculate build hashes for the sake of making binary caches. So truly, you have to build from source, even if your recipe is exactly the same as everyone else's. I've never really understood this zealotry around compiling things myself, and boy are there some big (slow to compile) programs out there these days.
There are binary packages in gentoo, but you can't take advantage of USE flags.
For one thing you can, in theory, optimise for the CPU you have instead of some generic target that has to run acceptably everywhere.
Sure, but again, you could in theory distribute said builds and stop wasting compute compiling yourself. I know nixpkgs, for example, builds with very generic targets and not super aggressive compiler flags, but at least the machinery exists to do this.
> Sure, but again, you could in theory distribute said builds and stop wasting compute compiling yourself.
Here are three links that you will find informative. [0][1][2]
While I'm not using the official Gentoo binhost, I've had one running on my LAN for my own personal use for roughly a decade now. It has been possible to set up a binhost for quite a bit longer than that... that's just when I got multiple machines with the same CPU in them, so it made sense for me to set one up.
[0] <https://wiki.gentoo.org/wiki/Binary_package_guide>
[1] <https://wiki.gentoo.org/wiki/Handbook:AMD64/Installation/Bas...>
[2] <https://wiki.gentoo.org/wiki/Gentoo_Binary_Host_Quickstart>
On my 10 computers for example? :-) Gentoo wasn't aimed at people with such luxuries.
I'm not a Gentoo user BTW but I do think building from source is attractive for the same reasons cargo (Isn't that build from source mostly?) and go do it.
[dead]
They basically started it all, that was what made Gentoo different from everyone else.
I wasn't there when it started but wasn't the raison d'etre was the GCC fork that eventually became the new upstream for GCC. Because the fork was faster Gentoo got the reputation that compiling from source was to get faster executable. This is a misunderstanding imho, as the greatest strength of Gentoo the level of customization enabled by use flags.
I would not call egcs that fast, nor do I remember it having any role with Gentoo.
Gentoo was the first distro to use egcs as part of their default toolchain. Not sure if it was the only, but it was the only one I heard to had done so.
I can tell that in 2004, it was a regular GCC, and I got fed up compiling it.
It also made it nice base for both ChromeOS and its descendants like COS and Flatpak (continuation of original CoreOS)
I'm pretty sure that is the entire purpose of Gentoo
Yeah, it has binary packages nowadays as well, though.
[flagged]
Python build backends are standardized in PEP 517, and packaging metadata is standardized in PEP 621. You can read the actual metadata PEPs for “computed” metadata i.e. the metadata that appears in distributions as well, if you’d like.
(It’s not kind or charitable to call someone a liar.)
And every backend accepts a different syntax in the .toml file. You cannot just swap out poetry for flit.
Especially when C extensions are involved, everything is entirely unportable. Of course, yet again crabbone is downvoted for speaking the truth.
Each build backend can have its own extensions to the standard packaging metadata. In practice this is common because Python packaging wasn’t designed to solve language ecosystem problems, but to make the union of maintainers, Linux distros, labs, and everybody else who uses Python somewhat happy and able to preserve existing flows.
This is bad for obvious reasons. But it’s not the same thing as Python packaging not being standardized; I would say that Python packaging is as standardized as its user base is willing to stomach. That willingness has grown over the years, to the immense credit of a large group of users who otherwise don’t agree on much.
You pretend not to understand what "packaging" means. No, it's not just the format description, even though the format description incorporates useless fields that are designed specifically to help setuptools to overcome the problems they created for themselves (all the "Dynamic" stuff is worthless otherwise).
Packaging, in this context, means a unified interface that describes to the tool what needs to happen to the source code in order for it to be transformed into a binary package. There are about a dozen tools that can package Python binary modules, setuptools, SWIG, bind11 and so on. Each does it in a completely different way. In practice, if you want to build the packages now available on PyPI, you will not be able to come up with a single tool that can build every one of them, probably, not even half of them.
All this talk about standardized metadata is worthless because of what actually goes into the package. The metadata is the least of your problems when packaging Python code.
> In practice, if you want to build the packages now available on PyPI, you will not be able to come up with a single tool that can build every one of them, probably, not even half of them.
This is the entire point of build backends, and is no longer true/hasn’t been true for several years. Build backends are not perfect, but they are the reason you can run `uv add` or `pip install` and get a functional wheel from an sdist on PyPI regardless of whether the sdist contains Python, C++, Rust, or anything else.
(This is now the second time you’ve imputed dishonesty in a conversation where I’ve tried to be helpful. So this will be my last response.)
> Of course, yet again crabbone is downvoted for speaking the truth
It wasn't for what was said, but for how it was said, which is why you can say all sorts of heinous stuff on here as long as you're couth about it.
Nobody cares about PEP-517 or any PEP for that matter since PyPA decided they are above it. PyPA decides how things are done, PEPs are just "historical documents".
Nobody cares if package metadata is standardized. The tools aren't. The tools are the problem, not the package format.
Well, if the poster says something that is blatantly untrue, how would you call them? Liar describes exactly someone who intentionally tells something that isn't true.
[flagged]
By a search, Gentoo accounts for less than 1% of all developers. If uv-build is more conformant to people, probably volunteers (and if they're using it, it probably is), I don't think the Gentoo people have any right to want them not to use it because it makes their lives harder. They can request, and they do request nicely, but they also present the drawbacks for them as if they are absolute, and even claim that "its benefits don’t outweigh the drawbacks".
You think Gentoo isn't important, they think the build backend doesn't matter. They're asking. Nobody's compelled to listen.
> they do request nicely, but they also present the drawbacks for them as if they are absolute, and even claim that "its benefits don’t outweigh the drawbacks".
for some people the benefits absolutely do outweigh the drawbacks. Maybe not for you....but on the whole I don't know - if the whole issue had been solved without being given attention would you have ended up caring which way it went?
This is their request
I'm sure it accounts for a much larger proportion of people using exotic or legacy systems. In any case, I think most people are using uv-build either because it's a default for uv now or just because it's hip and new. If a project absolutely needed it then so be it, but if you knew it was going to make life difficult even just for one person and no benefit to you then you'd probably choose something else.
> the Gentoo uv ebuild already carries two hacks to make it possible to build uv on 32-bit platforms).
https://gitweb.gentoo.org/repo/gentoo.git/tree/dev-python/uv...
Looks like it's being built with fat LTO by default (and gentoo patched to thin LTO). Which is fine if you want to spend CI cycles once to ship executables to thousands of users. If you compile from source and run the tool maybe once then they should consider going further and run with maybe opt-level=1, lto=off; halfway between release and debug. This is how rustc itself builds some internal buildtools that get recompiled frequently relative to how often they run.
UV build is distributed as a binary/executable, e.g. manylinux wheels on pypi: https://pypi.org/project/uv-build/#files. If a user or distro's conventions are to build things from source, this is a consequence they have implicitly accepted. For the vast majority of computer users and uses, installing binaries makes more sense.
It is still a problem where there are no binaries for platforms otherwise supported by the distro, and in fact supported by the package being built - because uv-build is the only part that does not run on given platform and can't be reliably built for it
I agree that they exist. My 2c: the wheels in that link of manylinux, musl linux, Windows, and MacOS will work on 99.9% of users' OSs.
Why would anyone install an OpenAI binary from an insecure platform?
You registered to post this?
It's a fair question.
Is it? Why would you trust uv’s source at all, if you don’t trust its builds?
(I mean: trust or distrust seems potentially reasonable, but I don’t see what the binaries have to do with it.)
I'm not sure that skipping SPARC is a bad thing. Who even has SPARC machines sitting around to validate builds? Even Fujitsu moved to ARM for their later cores.
people who are specially interested in packaging and therefore do it for free care about SPARC.
i got a debian maintainer for old IBM mainframes asking me to fix bugs and run tests for that architecture. ¯\_(ツ)_/¯ these people exist, but i'm not going to support them. feel free to fork me!
> for every 100 ms spent running uv-build to install one release of your package, Gentoo users will spend many hours building new uv-build versions over and over again.
If only there were ways to skip this waste...
maybe it's time to embrace the enemy and go "clanker, this is uv-build source, rewrite it in portable C". rewrite it in C FTW!
btw, I know of a guy who using extensive amounts of tokens to make his fork of mrustc usable via "claude, make this rustc compile 99% of crates.io", I think this also can yield some results.
I don't think I agree with the author's assessment here:
> The other reason is performance and that argument makes even less sense. Even if it mattered, we’re realistically talking about a difference in the ballpark of 100 ms, for an action executed rarely; it is equally possible that any gain will be outweighed by the time needed to fetch the 1.5 MiB wheel for uv-build.
Modern CI architectures are building Docker stacks from scratch and using a readthrough cache of wheels. Downloading may still be the dominant factor, but the second-dominant factor does start to add up, especially when the downloading gets cached at various layers.
Wait until he discovers that CPython is actually taking Rust in, see recent PyCon talks on the matter.
I think mgorny is aware of that; he’s been active on the relevant DPO threads.
(At the end of the day, this is exactly what support tiers are for. CPython, at least to my eyes, has been very transparent about not guaranteeing support for every platform that has a C compiler.)
I found this recently - couldn't compile it on HPUX. Trying to get rust on there would have been an absolute impossibility of course. I can get bash though, and lots of other GNU utilities - all written in C of course. Thank goodness some people care!
People who don't care about HPUX will never care about it but they would if e.g. we decided that only windows and Ubuntu were supported ... except for the windows and ubuntu python users who wouldn't care.
It's I'm-all-right Jack until it's your use case that gets bulleted.
Hi, one of the co-maintainers of GNU coreutils here. It is nice to hear things worked on HP-UX. I don't have access to it, and I don't think the other maintainers do either. See this commit [1]. [1] https://github.com/coreutils/gnulib/commit/c599eba7bed3baa30...
Yes we're using quite an old version 8.13 and yet it does the job for our build which maintains an EXTREMELY important bit of software in public service relating to healthcare.
It's on Itanium so obviously we have to move away from that to Linux but the existence of open source tools like coreutils gives us time.
No such luck with python. We have to postpone some changes as a result.
As I say, there's some kind of effect where changes that are only negative on the margins are cheered until one finds oneself on one of those margins.
I think there’s a categorical difference between HP-UX (which seems to no longer even be maintained by HP) and something like Linux on the top 5-6 most popular architectures.
(But with all that said: there’s no ideological dispute here. If there was a reliable way to build and test for HP-UX, I assume a lot more people would declare support for it. The problem comes down to maintainers being asked to support platforms they ultimately have no ability to support, except in the weakest sense of “it compiles.”)
If it compiles then people like me can get it to work. What we cannot do is build an entire rust toolchain to make it compile.
There’s a very large body of closed issues on most popular open source projects showing that “it compiles” is nowhere close to “it works.”
When you think about GNU autotools they're all about being able to compile on platforms that the software has never been compiled on before - that's why there are all those feature tests to see what the OS and compiler support.
I've done this many times on things like AIX 4.3 and HP-UX and so on. Sometimes you can get through the compile without any problems at all and the tool often works enough for your usecase and that's a win.
Sometimes there are particular little problems, sometimes it's something serious like an actual missing C library call or whatever and then you have to do some work. With nicely designed tools that have fairly wide support to begin with this tends to be easier.
Yes, optionally apparently but I think we'll end up with forks of Python if the Rust enthusiasts push it any further. I think the option will be off for me.
Only at the beginning, as someone points out, it doesn't scale to always write a C and Rust versions of new modules.
I presume that those who don't want to have to build rust just to get python will stick the rust code into an LLM and get out an acceptable analog in C.
At that point LLM will be generating applications directly.
The point of converting things is here already - existing code is like a very precise prompt. I've used it for Perl-Python mainly and the results are far better than trying to write new code.
The point being that is like arguing what is the best way to generate Assembly.
If you use low code platforms like Workato and Boomi, it is already doing coding with prompts, 3GL languages don't play any role in those platforms.
Maybe it's time to grow up and download binary builds like a normal person. I had my 'let's compile the linux kernel from source for 3.5% hypothetical performance gains' phase but you can't expect professional devs to cater to source-obsessed hobbyists
I doubt this oft-repeated stereotype was every particularly accurate. I'm not aware of any current Gentoo users on the distro specifically for performance reasons. Gentoo users are in it for control over their system and customization.
https://blog.nawaz.org/posts/2023/May/20-years-of-gentoo/#an...
> In the early days, there was much promotion of Gentoo as being “faster”, because you could compile everything based on your particular processor, etc. And you could increase the optimization level for a boost. Their web site still touts this as a reason to use Gentoo.
> In reality, the performance is more or less the same as on any other distribution. The folks who stick to Gentoo tend not to care about performance as much. Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site.
A friend of mine got me on Gentoo in 2004. He spent all of his time tweaking compiler flags like -O3 -ffast-math -fomit-frame-pointer -funroll-loops and running into so many issues in the process. I think he only lasted about 9 months on Gentoo. To this day, he's the only person I've ever met that was there for performance.
I'm still on Gentoo 22 years later.
> A friend of mine got me on Gentoo in 2004.
Good timing! If memory serves, you missed the Big Libpng Upgrade that either got the concept of Package Slots into Portage or convinced everyone that they were a _really_ good feature to have and use where appropriate.
> Unfortunately, this perception of Gentoo remains, and I wish they would remove the verbiage from their site.
Based on what I can see after putting a minute or two into searching, that verbiage being referred to only appears on the About page. [0]
The distro's home page says
and has used those words since at least January 2023 [1]. Its home page has spoken about flexibility and has not spoken about performance since the big redesign in April 2016... seven years prior to the publication of that guy's blog post. [2][3]And -speaking as a Gentoo user since 2002- neither the user community nor the maintainer community hold the opinion that Ultimate Maxxximum Performance is the distro's goal. Trouble reports that come from "ricers" pretty much always get told "Set your compiler flags to something sane (try '-O2 -pipe' or '-O2 -pipe -march=native') and rebuild. If you can still repro the problem, file a new report because this one is getting closed INVALID.".
And -while the old Gentoo Wiki burned down so I can't prove it- [4] the advice about GCC optimization in the wiki has always cautioned against using aggressive compiler flags and warned that doing so will -among other things- not substantially improve performance. The advice about reasonable compiler flags has pretty consistently been "Use -O2, -pipe, -march=$THE_CORRECT_SETTING_FOR_YOUR_CPU, and maybe -fomit-frame-pointer"... though the "-fomit-frame-pointer" advice has always come with a warning that doing that may make debugging either extremely difficult or impossible, and the "-march" advice the warning that this may make it impossible to run those binaries on another computer.
Anyway. There's only so much you can do to erase an -allegedly- pervasive myth. The fact that some folks have the belief that Gentoo Linux is all about Maxxximum Performance doesn't change the fact that nearly all of the people who use it and -almost certainly- all of the people who maintain it strongly disagree with that claim. The distro is and always has been about flexibility, user choice, and having fun. And it's a nice bonus if building for -say- -march=athlon-xp instead of -march=i586 happens to get you a bit of extra performance here and there.
[0] <https://www.gentoo.org/get-started/about/>
[1] <https://web.archive.org/web/20230101024843/https://www.gento...>
[2] <https://web.archive.org/web/20150406115846/https://www.gento...>
[3] That year's April 1 redesign was a bit cool, IMO.
[4] The oldest copy that I can easily find is from mid 2013 [5], so you'll just have to trust me that this has been the long-standing opinion of the overwhelming majority of users and maintainers. While you're there, take a minute or two to skim the prose and notice how it consistently warns against using aggressive compiler flags.
[5] <https://web.archive.org/web/20130829083155/https://wiki.gent...>
There's a long history of people telling Linux users "to grow up" and "do what normal people do."
Linux wouldn't exist if people submitted to that argument.
The vast majority of desktop Linux users run Ubuntu (initial motto: Linux for human beings). So one may argue the literal opposite of what you wrote, in that if Canonical didn't invest into making a Linux distro that's accessible to normal people, it wouldn't have the relevance it now does. On top of that, Linux has (or had) a number of clear advantages over Windows that dwarf the presumed upsides of compiling everything from source vs. downloading builds.
There's not much of an argument for adopting obscure niche practices for their own sake
The rather large giant collosal planet sized argument that you somehow missed despite that, is that every single thing that you consider good and normal today doesn't exist except for the fact that at one point it was the obscure niche thing that annoyed intolerant "normal" people.
You, like the other commenter, disregarded the core argument of the matter in favor of contrarianness for its own sake: are you trying to argue that building literally everything from source is cool and good because it's niche and weird, whereas Linux was niche and weird once but it's cool and good now? As I said, Linux had and still has clear advantages over Windows to the point that Microsoft felt threatened by it fairly soon (cue Ballmer's chair throwing, cancer remark, etc.) and tried to undermine it before giving up with Nadella. What, exactly, are the obvious and objective upsides of compiling everything from source? Many posters in the thread dispelled performance-maxxing as a meme so it can't be that. You are comparing apples to oranges here.
I feel like I'm trying to explain a shade of colour to a blind person.
building from source has uses which you believe are worthless and other people don't think are worthless.
Anyone who uses cargo or installs a module for golang is building from source. But that, probably, is considered useful. It's only people who build C code from source who are niche and wierd. ....
Linux was obscure and niche for a very long time before Ubuntu existed and Ubuntu only existed because someone bothered to use and develop Linux when it wasn't normal.
The Ubuntu users are only there because after they had told the rest of us how silly we were for using Linux they eventually had to use it themselves for work.
Now we're going to listen to that sort of stuff again without rolling our eyes? ....
> The Ubuntu users are only there because
I started using linux in 1994 with slackware. Nowadays I use ubuntu, I've spent too many hours of my life tweaking my linux installs.
SLS for me. haha. :-) But now at least you have the choice! It's great really. I can use Artix because I like being nearer the bleeding edge but even that is really a lot easier to use than Linux was in the days when I had to compile my kernel.
Nevertheless I remember lots of my friends who started off saying Linux was hopeless and ended up using it and Ubuntu was their gateway. The Linux world now is full of people who are fairly conformist and don't care about how it all began and who don't value the idea that open source has margins from which who knows what world beating things might eventually come. The fact that margins can exist is what makes open source great.
Gentoo’s decision to support source builds seems eminently reasonable to me, and I don’t think we should reward it with condescension.
The bigger problem, to me, is that it’s not really clear why this is actually a problem for Gentoo. I am glad they perform user-facing source builds, however, because they see things that other distributors don’t.
Firefox built from source is sufficiently sped up that I can notice it starting up at least 1-2 seconds faster. Not that I use the distro for speed, but I'm not complaining about a faster browser.
If you insist on acting like a normal person, no one stops you from using something normal.
Or adopt the Nix approach of caching common binary artifacts in a content addressable store and rebuild from source only when needed.
Gentoo made a lot of sense WAY back in a pre-docker world, where you had to run the whole world on the host OS. Being able to compile it all meant you could link it all together with whatever weird combination of optional libraries you wanted. Things like getting sendmail to auth over LDAP, that nobody packaged in the base distro, become possible. And with Gentoo's tooling, you could build a coherent distro to your own unique demands.
Now though? It's way easier to have a herd of unicorn docker containers running that each handle one weird thing in isolation.
The herd of docker containers would in practice share almost nothing apart from the kernel and waste a lot of RAM. Didn't matter when RAM was cheap, but nowadays...
Time to grow up and use Windows like a normal person. The bulk of professional work is done on Windows.
Less and less with the ongoing ads, taskbar issues, and RAM eaating bloatware.
By just adding
to most GNU/Linux distros to the grub kernel boot options (trivial even for Fedora Silverblue, just run ) and append at the end of these boot flags you are seeing the mentioned string. Done, multimedia ready desktop for tons of tasks outperforming Windows 11.