> Technical superiority doesn't win ecosystem wars. Linux won through a combination of fast decisions, the viral GPL licence, and strong enterprise backing from Red Hat and IBM. Then Google, Facebook, and Amazon happened — hungry for datacenters, developing tools to manage growing infrastructure at scale. They set the direction for the entire industry.
In the mid 1990's the hardware driver support on Linux was much broader.
Copy / paste of my comment from last year about FreeBSD
I installed Linux in fall 1994. I looked at Free/NetBSD but when I went on some of the Usenet BSD forums they basically insulted me saying that my brand new $3,500 PC wasn't good enough.
The main thing was this IDE interface that had a bug. Linux got a workaround within days or weeks.
The BSD people told me that I should buy a SCSI card, SCSI hard drive, SCSI CD-ROM. I was a sophomore in college and I saved every penny to spend $2K on that PC and my parents paid the rest. I didn't have any money for that.
The sound card was another issue.
I remember software based "WinModems" but Linux had drivers for some of these. Same for software based "Win Printers"
When I finally did graduate and had money for SCSI stuff I tried FreeBSD around 1998 and it just seemed like another Unix. I used Solaris, HP-UX, AIX, Ultrix, IRIX. FreeBSD was perfectly fine but it didn't do anything I needed that Linux didn't already do.
> FreeBSD was perfectly fine but it didn't do anything I needed that Linux didn't already do.
That's pretty much it. A lot of the people I see using a BSD these days do so because they always have and they prefer what they know, which is fine, or they just want to be contrarian.
Realistically, aside from edge cases in hardware support, you can do anything you want on any modern *nix. There's not even as much of a difference between distros as people claim. All the "I want an OS that gets out of my way" and similar reasons apply to most modern well-maintained distros these days. It's more personality and familiarity than anything objective.
I went from Slackware in 1994 to Red Hat in 1998 to Fedora when they split into Fedora and RH Enterprise. Every 2 or 3 years I will install a different distro in a VM and see "Okay, now I see what it's about." But I have no interest in switching as long as Fedora does everything I need. I don't really understand the people that distro hop. I just assume they are really young and I have work to do and a family to take care of.
I get that. I stopped using Slack around...not sure, maybe 2007 or so when I tried to do my normal minimal setup and mplayer wouldn't run without Samba installed, and the community was hostile to anyone who didn't do the recommended full install. I never wanted it a a feature complete desktop but that's the market they tried to focus on.
Took me a while to settle on Alpine after trying Arch and Void, but I can't imagine why I would ever leave unless they change something drastic.
> FreeBSD was perfectly fine but it didn't do anything I needed that Linux didn't already do.
I broadly agree, even as a FreeBSD fan myself; things have converged a lot over the decades. But still, I generally feel that while you can get the same work done in both, FreeBSD does things better (and/or cleaner, more elegant, etc) in many cases.
The overall feeling of system cohesion makes me happier to use it, from small things like Ctrl-T producing meaningful output for all the base OS tools, to larger and more amorphous things like having greater confidence core systems won't change too quickly over time (eg: FreeBSD's relatively stable sound support, versus Linux's alsa/pulse/pipewire/..., similar for event APIs, and more).
Though I totally feel your pain about latest-and-greatest hardware driver support. Has gotten better since the '90s, but that gap will probably always be there due to the different development philosophies.
I hope FreeBSD never gets too "Linux-y"; it occupies it's own nice spot in the spectrum of available options.
My feeling at the time was that BSD developers and many users were at least 30 years old or much older and had professional jobs and money.
The Linux community felt like college students with no job and not much money. That included Linus Torvalds himself who developed the kernel while in college and wasn't rich. DEC basically gave Linus an Alpha to get him to port the kernel to it.
> To solve the distribution and isolation problem, Linux engineers built a set of kernel primitives (namespaces, cgroups, seccomp) and then, in a very Linux fashion, built an entire ecosystem of abstractions on top to “simplify” things: [...] Somehow we ended up with an overengineered mess of leaky abstractions
Not sure I like the value judgement here. I think it's more of a consequence of Linux' success. I am convinced that if it was reversed (Linux was niche and *BSD the norm), then a ton of abstractions would come, and the average user would "use an overengineered mess" because they don't know better (or don't care or don't have a need to care).
Not that I like it when people ship their binary in a 6G docker image. But I don't think it's fair to put that on "those Linux engineers".
I don't agree with that. FreeBSD has more of an engineering than a hacking mentality and it shows in the various architectural choices.
And containers really are a VM-light, so you might as well use the real thing, in fact, VMWare for a long time thought that their images would be a container like thing and many larger installations used them as such.
I don't think it's necessarily true, compare the BSD utils to the GNU utils and the style difference is very visible.
On the other hand, I don't think the comparison between jails and docker is fair. What made Docker popular is the reusability of the containers, certainty not the sandboxing which in the early days was very leaky.
Indeed, that Docker is functionally a cross-distro rolling release package manager, configuration standard, and service supervisor[1] is the appeal to me. Any isolation it achieves is necessary for that to all work reliably, but is not why I use it.
Inability to find a service I want to run on Github and 95+% of the time to be able to configure it and have it running and fully managed with usually just a one-liner shell script like 10 minutes later just by finding an existing docker image is the thing I’d lose with jails. That’s all of the value of docker to me personally. Jails could be a building block toward that, but last I checked there’s no deep and up-to-date library of “packages” I can reach for, using jails, which makes it pretty much useless to me.
1: I have like eight or nine services running on my home Debian system, they all auto-restart and come back up on reboot, and I’ve not had to touch Systemd once on that machine.
I do not know which is the difference, but you really feel a difference.
It might be of homogeneity, i.e. the FreeBSD tools behave in a consistent way, while there are significant differences between the Linux tools, depending on which were the opinions of their particular authors about how the traditional UNIX tools should be changed.
For instance, at some point in time, long ago, in Linux the traditional "ifconfig" and a few related commands have been replaced by "ip", for managing networking.
The Linux "ifconfig" needed an upgrade, as it could do only a small fraction of what the FreeBSD "ifconfig" could do. Nevertheless, until today, decades later, I have been unable to stop hating the Linux "ip".
I cannot say why, because in other cases when some command-line or GUI utility that I had used for many years was replaced by an alternative I instantly recognized that the new UI was better and I never wanted to use the old UI again.
So while both FreeBSD and Linux have started with the same traditional UNIX utilities, they have evolved divergently and now they frequently feel quite differently, in the sense that the various options in commands or in configuration files may match your expectations only when taking into account the identity of the OS. Overall FreeBSD has been more conservative, but there are also cases when it has made bigger changes, but such changes seem more carefully planned and less haphazard than in the Linux world.
And for the whole world, too. I don't need to build my own local stripped down version of Alpine Linux with python, somebody's already dike that for me.
I frequently see freeBSD jails as a highlighted feature, lauding their simplicity and ease of use. While I do admire them, there are benefits to the container approach used commonly on linux. (and maybe soon freebsd will better support OCI).
First it's important to clarify "containers" are not an abstraction in the linux kernel. Containers are really an illusion achieved by use of a combination of user/pid/networking namespaces, bind mounts, and process isolation primitives through a userspace application(s) (podman/docker + a container runtime).
OCI container tooling is much easier to use, and follows the "cattle not pets" philosophy, and when you're deploying on multiple systems, and want easy updates, reproducibility, and mature tooling, you use OCI containers, not LXC or freebsd jails. FreeBSD jails can't hold a candle to the ease of use and developer experience OCI tooling offers.
> To solve the distribution and isolation problem, Linux engineers built a set of kernel primitives (namespaces, cgroups, seccomp) and then, in a very Linux fashion, built an entire ecosystem of abstractions on top to “simplify” things.
This was an intentional design decision, and not a bad one! cgroups, namespaces, and seccomp are used extensively outside of the container abstraction. (See flatpak, systemd resource slices, firejail). By not tieing process isolation to the container abstraction, we can let non-container applications benefit from them. We also get a wide breadth of container runtime choices.
I still see FreeBSD as being great for things like networking devices and storage controllers. You can apply a lot of the "cattle vs pets" design one level above that using VMs and orchestration tools.
If you don't want to use the base system (which docker is NOT the base system on Linux) then Bastille offers a pretty much identical workflow to docker, but built on FreeBSD jails: https://github.com/BastilleBSD/bastille
> I don’t know why i keep hearing about jails being better
Jails have a significantly better track record in terms of security.
I can delegate a ZFS dataset to a jail to let the jail manage it.
Do Linux containers have an equivalent to VNET jails yet? With VNET jails I can give the jail its own whole networking stack, so they can run their own firewall and dhcp their own address and everything.
You've been able to setup separate firewalls, network interfaces, IP addresses, etc. for probably 20 years using network namespaces. How do you think container networking is implemented? But you can also use it through other tools; for example, I use firejail to isolate a couple of proprietary desktop applications such that they cannot contact anything on my desktop (or network in general) except the internet gateway.
Is there a docker-compose analogue in Bastille? I like being able to spin up an isolated local copy of my infrastructure, run integration tests, and then tear it all down automatically. I'd like to be able to do a similar thing with jails. I wonder if there's a straightforward way to achieve something similar with VNET jails?
Not that I'm aware of. FreeBSD did recently gain support for OCI containers and therefore has podman. I see podman-compose is in the ports tree, but I haven't tried it myself.
The jails vs containers framing is interesting but I think it misses why Docker actually won. It wasn't the isolation tech. It was the ecosystem: Dockerfiles as executable documentation, a public registry, and compose for local dev. You could pull an image and have something running in 30 seconds without understanding anything about cgroups or namespaces.
FreeBSD jails were technically solid years before Docker existed, but the onboarding story was rough. You needed to understand the FreeBSD base system first. Docker let you skip all of that.
That said, I've been seeing more people question the container stack complexity recently. Especially for smaller deployments where a jail or even a plain VM with good config management would be simpler and more debuggable. The pendulum might be swinging back a bit for certain use cases.
- Stable OS coupled with rolling packages. I am on the previous FreeBSD version (14.3-RELEASE, while 15 is out) but I have the very latest KDE.
- A ports collection where you can recompile packages whenever you're not happy with the default settings. Strict separation between packages and core OS. Everything that is from packages is in /usr/local (and this separation is also what makes the above point possible).
- ZFS on root as first-class citizen. Really great. It has some really nice side tooling too like sanoid/syncoid and bectl (the latter is part of the core OS even).
- jails for isolation (I don't really use it like docker for portability and trying things out)
- Clear documentation because there are no different distros. Very good handbook. I like the rc.conf idea.
- Common sense mentality, not constantly trying to reinvent the wheel. I don't have to learn a new init system and I can still use ifconfig. Things just work without constantly being poked around.
- Not much corporate messing around with the OS. Most of the linux patches come from big tech now and are often related to their goals (e.g. cloud compatibility). I don't care about those things and I prefer something developed by and for users, not corporate suits. No corporates trying to push their IP onto the users (e.g. canonical with their Mir, snaps etc)
- Not the same thing as everyone else has. I'm not a team player, I hate going with the flow. I accept that sometimes comes with stuff to figure out or work around.
ZFS boot+root on Linux is amazing as well. It's kind of sad to see Linux Mint has moved away from supporting this in their installer, but it probably could still be done manually I guess. After upgrade, if something goes wrong? zfs rollback both to a snapshot made just before the upgrade and reboot.
I don't use it full time, only in a VM, but all these things sound positive to me.
*everything. I've really been using it since 4.x. Imagine this: being able to upgrade a system in-place with freebsd-update from minor to major to minor version without everything breaking or having to say a prayer before. And that's just one thing I love about it. Clear separation of userland (/usr/local/etc), rock-solid stability in networking, zfs on root.
I had to do 'bonded' interfaces on Debian the other day. It's what, 5 different config files depending on which 'network manager' you use. In FreeBSD it's 5 lines in /etc/rc.conf and you're done.
And don't even get me started on betting which distribution (ahem CentOS) will go away next.
I actually laughed out loud. Try upgrading CentOS to Rocky vs FreeBSD 11 to 15 ( that's FOUR major versions from 2017 I think ), and tell me again how good it is.
In LTS environments where I need to upgrade OS's, FreeBSD is a no-brainer.
> I actually laughed out loud. Try upgrading CentOS to Rocky vs FreeBSD 11 to 15 ( that's FOUR major versions from 2017 I think ), and tell me again how good it is.
I laughed out loud, there is no in-place upgrade mechanism for that in those distros and never has been, that is the nature of those distros. They release patch/security updates until they go EOL, which is measured in units closer to decades than years.
I don’t have a problem with BSDs. That’s cool you like upgrading in place.
The best and most laugh-inducing part of your whole point is that centos now not only allows you to do in-place upgrades, that’s the whole fucking point.
So then what's the point of mentioning Rocky as CentOS's successor ? In what way is it 'succeeding' ? That you can do a fresh install of Rocky ? And those stuck on CentOS can't upgrade ? Really useful those decades of support if your distro goes belly up
I'm using either Docker Compose or Docker Swarm without Kubernetes, and there's not that much of it, to be honest. My "ingress" is just an Apache2 container that's bound to 80/443 and my storage is either volumes or bind mounts, with no need for more complexity there.
> The jails vs containers framing is interesting but I think it misses why Docker actually won. It wasn't the isolation tech. It was the ecosystem: Dockerfiles as executable documentation, a public registry, and compose for local dev. You could pull an image and have something running in 30 seconds without understanding anything about cgroups or namespaces.
So where's Jailsfiles? Where's Jail Hub (maybe naming needs a bit of work)? Where's Jail Desktop or Jail Compose or Jail Swarm or Jailbernetes?
It feels like either the people behind the various BSDs don't care much for what allowed Docker to win, or they're unable to compete with it, which is a shame, because it'd probably be somewhere between a single and double digit percent userbase growth if they decided to do it and got it right. They already have some of the foundational tech, so why not the UX and the rest of it?
I think Jails started as a tool of it's time, it's about the same thing as virtualization in making isolated systems when dependencies start to diverge, but aimed at the issues of sysadmins that had to manage their own systems, not a quick developer experience.
Even if "jailsfiles" were created the ecosystem would need to start from scratch and sometimes it feels like people in the FreeBSD ecosystem have a hard enough time keeping ports somewhat up to date, let alone create something new.
Luckily Podman seems to support FreeBSD these days for docker images, but the Linux emualation might be a bit of a blocker so not a 100% solution.
Docker's client/server design also allowed for things like Docker Desktop, which made the integration seamless with non-linux systems. Jails have nothing like that, so the only system that will ever run jails is FreeBSD. Also, I'm not up to speed enough to know, but do jails even have a concept of container images?
Plus a script to unpack the tarball somewhere and launch some entry point in a jail. Not conceptually hard, but the OCI spec has a bit more to it than that, and now we're into "write dropbox with rsync" territory...
I did some looking around, and I see that ocijail is a thing, so that's probably what I was looking for.
> Jails solve the isolation problem beautifully, but they don't have a native answer to shipping. That gap is real, and it's one of the main reasons the ecosystem around jails feels underdeveloped compared to Docker's world.
The link literally uses the term ecosystem. Several times actually.
I've tried this ... I've haven't got much mileage on this, sadly.
Many Linux syscalls are unemulated and things like /proc/<pid>/fd/NN etc are not "magic symlinks" like on Linux so execve on them fails (i.e there is rudimentary /proc support, it's not full fleshed out).
TL;DR Linux containers on FreeBSD via the podman + linuxulator feel half baked.
For example, try using the alpine container... `apk upgrade` will fail due to the /proc issue discussed above. Try using the Fedora container `dnf upgrade` will fail due to some seccomp issue.
The future of containers on FreeBSD is FreeBSD OCI containers, not (emulated) Linux containers. As an aside, podman on FreeBSD requires sudo which kinda defeats the concept but hopefully this will be fixed in the future.
I ran a whole company on top of FreeBSD back in the day (2005 ish). It was great, and ran all my personal pcs the same way (hell, refusing to install windows to try out this bitcoin idea is even now a good idea).
But somehow Linux still took over my personal and professional life.
Going back seems nice but there need to be a compelling reason -docker is fine, the costs don’t add up any more. I do t have a real logical argument beyond that.
In the early years after 2000, FreeBSD 4 had a much better performance and reliability in any networking or storage applications in comparison with the contemporaneous Linux and Windows XP/Windows 2000.
However, in 2003 Intel introduced CPUs with SMT and in 2005 AMD introduced multi-core CPUs.
These multi-threaded and/or multi-core CPUs quickly replaced the single-threaded CPUs, especially in servers, where the FreeBSD stronghold was.
FreeBSD 4 could not handle multiple threads. In the following years Linux and Windows have been developed immediately to take advantage of multiple threads and cores, while FreeBSD has required many years for this, a time during it has become much less used than before, because new users were choosing Linux and some of the old users were also switching to Linux for their new computers that were not supported by FreeBSD.
Eventually FreeBSD has become decent again from the PoV of performance, but it has never been again in a top position and it lacks native device drivers for many of the hardware devices that are supported by Linux, due to much fewer developers able to do the necessary reverse engineering work or the porting work for the case when some company provides Linux device drivers for their hardware.
For the last 3 decades, I have been using continuously both FreeBSD and Linux. I use Linux on my desktop PCs and laptops, and in some computational servers where I need software support not available for FreeBSD, e.g. NVIDIA CUDA (NVIDIA provides FreeBSD device drivers for graphic applications, but not CUDA). I continue to use FreeBSD for many servers that implement various kinds of networking or storage functions, due to exceptional reliability and simplicity of management.
The FreeBSD threading was perhaps behind but in general, but the big things in Linux VS FreeBSD was always the 4.3 licensing lawsuits that gave Linux a momentum that BSD never caught up with.
The real difference during that early 00s was that momentum bought 2 things that made FreeBSD a worse choice (and made even more people end up using Linux):
1: "commercial" support for Linux, firstly hardware like you mentioned, but in the way that you could buy a server with some Linux variant installed and you knew that it'd run, unless you're an CTO you're probably not risking even trying out FreeBSD on a fresh machine if time isn't abundant.
Also software like Java servers comes to mind, came with binaries or was otherwise easy to get running on Linux, and even with FreeBSD's Linux layer VM's things like JVM and CLR often relied on subtle details that makes it incompatible with the Linux layer (tried running .NET a year or two ago, ran into random crashes).
2: a lot of "fresh" Linux developers had a heavy "works on my machine" mentality, being reliant on Linux semantics, paths or libraries in makefiles (or dependencies on things like systemd)
Sure there is often upstream patches (eventually) or patches in FreeBSD ports, those last are good for stable systems, but a PITA in the long run since stuff doesn't get upstreamed properly and you're often stuck when there is a major release and need to figure out how to patch a new version yourself.
Yeah, I have a similar situation; FreeBSD is a great operating system, but the sheer amount of investment in Linux makes all the warts semi-tolerable.
I'm sure some people have a sunk-cost feeling with Linux and will get defensive of this, but ironically this was exactly the argument I had heard 20 years ago - and I was defensive about it myself then.. This has only become more true though.
It's really hard to argue against Linux when even architecturally poor decisions are papered over by sheer force of will and investment; so in a day-to-day context Linux is often the happy path even though the UX of FreeBSD is more consistent over time.
I know this comment is effectively a side tangent on a side tangent. but that was always the strangest thing to me as well. I remember in 2012 when I was debating fiddling around with Bitcoin. that was one of the things that turned me off. I was sure that there was no way something as brilliant as this was supposed to be was developed by windows user.
Which surely says something about all these ideological purity tests
Windows developers (like sysadmins) are of two kinds in my experience.
People who don't understand shit about how the system behaves and are comfortable with that. "I install a package, I hit the button, it works"
.. and
People who understand very deeply how computers work, and genuinely enjoy features of the NT Kernel, like IOCP and the performance counters they offer to userland.
What's weird to me is that the competence is bimodal; you're either in the first camp or the second. With Linux (+BSD/Solaris etc;) it's a lot more of a spectrum.
I've never understood exactly why this is, but it's consistent. There's no "middle-good" Windows developer.
The (install package, press button, it works) is great when you just want a boring OS since the interest is elsewhere rather than an itch on making the machine as perfect extension of onself.
The machine and installation is just fungible.
I think I've had Linux as a primary OS 2 times, FreeBSD once and osX once, what's pulled me back has been software and fiddling.
I'm on the verge of giving Linux or osX another shot though, some friends has claimed that fiddling is virtually gone on Linux these days and Wine also seems more than capable now to handle the software that bought me back.
But also, much of the software is available outside of Windows today.
Unix is easier to understand than the NT mess and everything it's in the open and documented, so you can achieve a good level of knowledge in the middle. OTOH in order to understand NT deeply you must be a reverse engineer. Also, on the other side, crazy experts under Wine (both ways, Unix and NT) OpenBSD and 9front do exist on par of these NT wizards. It just happen with Unix/9f you climb an almost flat slope (more in the second) due to the crazy simple design, while with NT the knowledge it's damn expensive to earn.
With 9front you OFC need expertise on par of NT but without far less efforth. The books (9intro), the papers, CSP for concurrency... it's all there, there's no magic, you don't need ollyDBG or an NT object explorer to understand OLE and COM for instance.
RE 9front? Maybe on issues while debugging, because the rest it's at /sys/src, and if something happens you just point Acid under Acme to go straight to the offending source line. The man pages cover everything. Drivers are 200x smaller and more understandable than both NT and Unix.
Meanwhile to do that under NT you must almost be able to design an ISA by yourself and some trivial compiler/interpreter/OS for it, because there's no open code for anything. And no, Wine is not a reference, but a reimplementation.
That's kinda true for older/integrated parts of Windows, lots and lots of functionality that people have come to rely on over the years, but also huge black-boxes that you need to not be intimidated at probing into to solve weird issues (that often becomes understandable if you have enough experience as a developer to interpret what the API surface tells about the possible internal implementation).
Yeah, both VServer and Virtuozzo were roughly contemporaneous with the initial Xen release (2001-ish?), and only a few months behind jails.
But Virtuozzo was hampered by being non-free - OpenVZ wasn't open sourced until 2-3 years later, by which time the damage had been done (but, of course, Xen headed in the opposite direction at roughly the same time!)
And Linux-VServer was held back by being focussed so directly at virtual hosting providers - it positioned itself against fcgi-suexec, fcgid, and php-fpm (and was much more unwieldy than any of them) rather than jails or VZ.
Both were more or less ignored until the late 2000s, by which time LXC had taken a lot of mindshare - allowing the "FreeBSD was years ahead with jails" meme to take root.
Is there any technical writeup which explains how the isolation exactly works, on containers and VMs? I have always heard the high level arguments of weak isolation, same kernel, etc but never the implementation details.
Long emotional rant ahead, you have been warned. The poorly thought out adoption of parts of systemd, and in particular systemd-oomd, are making me yearn for FreeBSD.
For all my computing career, I'd use Unix-alike because they let me develop software by having an idea, write some code, let it run in the terminal, chain things together, and see where it failed, and iterate. Terminals let this happen much faster than clicky GUI software, and contain a log of what was happening in a terminal pane (or gnus screen or tmux pane, because usually this is happening on big servers and compute clusters rather than my terminal/laptop)
I could launch a bunch of panes, have several lines of investigation going, and come back to it a day or week later when I had time because the terminal kept a log of everything that happened.
And a couple years ago I started noticing that things I thought I had launched would start disappearing. At first I thought I had started accidentally mispressing a key and killing a tmux pane rather than disconnecting.
But no! When I finally went back to a in-person Linux workstation and saw it happen to the entire terminal window, I knew something major had changed. It turns out that something called systemd-oomd was added to a bunch of distributions that now kills only entire cgroups of processes at a time, rather than a single process offender.
So now if you want to run processes and isolate the kill zone of a process, you have to wrap every freaking subprocess in an entire systemd-run wrapper or docker wrapper. And systemd-run won't work from many contexts, such as inside a Jupyter kernel.
Major breaking changes on fundamental system behavior are a huge problem these days. It's one thing to let the OS kill processes more when there's a memory issue, fine, great, go ahead. But why kill all the lightweight processes that could give feedback to a user?! And why force non-portable process launching semantics, that aren't even consistent across the entire system?!? So infuriating.
I was looking at TrueNAS CORE to see if it was a viable way to bsd-jail Linux containers. I'm really only doing this to get some protection from supply chain attacks given I'm fairly promiscuous at git-clone-and-run-a-build. Before that I was aiming for the same with Bastille and had got to the give up stage because it felt too fiddly to set up. This was a year ago. Maybe its better now
I am not quite sure what this means. I had a jail a few years ago and I remember there was a utility to "back" the jail up so you could put it on another system. Are there constraints with that utility. It seemed to work, maybe I am forgetting something ?
In any case I still think Jails are much better than the things Linux has. To me, it is creating a jail that is more difficult. There were ports that made it easier, I used one of them, but that port was abandoned at some point. I think it was "ezjail".
I switched my startup’s whole infra to FreeBSD a couple months ago. Found a use after free bug that Linux’s memory management was just fine with in Gnome XSLT lib that FreeBSD properly refused. Other than that smooth sailing, jails work great.
After IBM destroyed CentOS, all the Xorg politics nonsense, the list goes on with Linux, not interested. I just want something quiet and boring and stable and correctly designed. NetBSD would be my first choice but they don’t get the $ they need for drivers.
Done the same since 2018 circa, never looked back.
For a while even used it on the desktop, but was too much trouble due to specific tools we need that weren't supported properly. so we're using Linux on the desktop.
FreeBSD is stable, lightweight, gets out of the way, and without drama.
I do follow the news cycle and if I’m hearing about a software package in it, something is wrong with the people making the software and I don’t trust them. Software is an engineering discussion or at least it’s supposed to be. Here’s my community guidelines: everyone be nice and respectful engage in good faith and focus on the math. Being social is fine so long that it doesn’t become a diversion from the engineering discussion. We’re talking about code not a philosophical treatise. There are civil ways to settle disagreements. I’m so sick to death of the politics.
The engineers I’ve worked with my whole career had no problem getting the work done without getting into giant political debates. If your politics comes before the engineering I don’t want anything to do with it.
Really the whole theme that (from the article) "FreeBSD ships as a complete, coherent OS" is belied by this kind of nonsense. No, it's not. Or, sure, it is, but in exactly the same way that Debian or whatever is. It's a big soup of some local software and a huge ton of upstream dependencies curated for shipment together. Just like a Linux distro.
And, obviously, almost all those upstream dependences are exactly the same. Yet somehow the BSD folks think there's some magic to the ports stuff that the Linux folks don't understand. Well, there isn't. And honestly to the extent there's a delta in packaging sophistication, the Linux folks tend to be ahead (c.f. Nix, for example).
The key thing is that on freebsd you do not risk bricking your system by installing a port. Even though this guarantee has become less true with PkgBase
> The key thing is that on freebsd you do not risk bricking your system by installing a port
What specifically are you trying to cite here? Which package can I install on Debian or Fedora or whatever that "bricks the system"? Genuinely curious to know.
I was referring to the need to be careful to not modify/update packages also used by the base system. Since all packages are treated the same on Linux, you often can't tell which package can put you in trouble if you update it along with the dependencies it drags with it.
This kind of problem happens frequently when users add repositories such as Packman on Linux providing dependencies versions different from the ones used by the base system of the distro.
Experienced people know how to avoid these mistakes, but this whole class of problem does not exist on FreeBSD.
> Since all packages are treated the same on Linux
This is no longer the case in "immutable" distros such as Bluefin/Aurora, which uses ostree for the "base" distro, while most other user packages are installed with homebrew. Nix and Guix solve it in a very different way. Then there's flatpak and snap.
A lot of poor *BSD advocacy likes to deride Linux for its diversity one moment, then switch to treating it as a monolith when it's convenient. It's a minority of the users for sure, but they naturally make an outsized share of the noise.
I think you missed the point in my original comment. I explained I moved my platform with all dependencies and had 1 bug which was actually a silent bug in Linux.
In other words, it works. Your particular stack might have a different snag profile but if I can move my giant complex app there, yours is worth a shot.
FreeBSD is more complete than you make out. They also have hard working ports maintainers.
Well, sure, but that's a ridiculous double standard. You're making the claim (or implying it, at least) that FreeBSD is fundamentally superior because it's a unified piece of software shipped as a holistic piece of artifice or whatever. And that by inferrence it's unlike all that kludgey linux stuff that you can't trust because of politics or whatever.
But your evidence that it's actually superior? "it works". Well, gosh.
You'll tar the competition with all sorts of ambiguous smears, but all you ask from your favorite is... that you got your app to work?
At one time this was an interesting comparison... but now Linux has gotten so much development that even if FreeBSD was Betamax and Linux VHS (in the past)... I would say that Linux is now DVD ... and FreeBSD still remains betamax.
Don't get me wrong, FreeBSD is simple, elegant, consistent and well manicured. It seems to have picked up some pace again. I'm rooting for it.
> FreeBSD is worth a brief aside here, because it differs from Linux in a fundamental way. Linux is a kernel. What most people call "Linux" is actually that kernel combined with a GNU userland, a package ecosystem, and a set of choices that vary from distro to distro — Ubuntu, Fedora, and Arch are all running the same kernel but are meaningfully different systems underneath.
It is not incorrect but ... do people really care about that distinction?
Because in most situations I know of, when people refer to Linux, they almost never refer to the linux kernel. They refer to the whole operating system stack, which is typically put down via a distribution. So, Fedora, Gentoo, Arch, and so forth, are all "kind of" Linux. Barely anyone refers to the linux kernel if you look at all the discussions on the world wide web.
> FreeBSD ships as a complete, coherent OS
The BSDs often promote that aka "Linux is chaos, we are coherent and consistent operating system following intelligent design". Well ... this is the rise of worse is better, repeated: https://dreamsongs.com/WorseIsBetter.html
It is a great analogy that works on so many levels. Broken down to Linux versus the BSDs, I think 500 out of 500 top supercomputers running Linux kind of show which philosophy is better. The one that works better. That does not mean the BSDs are useless, but I am getting tired of the promo used by the BSD as "we are order, Linux is chaos". I compare this more to Lego building blocks. With Linux there is a stronger focus on having building blocks available. You can build up things. You have projects such as LFS/BLFS (Linux from scratch). The BSDs do not have something comparable. Which operating system is the better tinker OS? Which community created git? (Ok ok that was Linus so not really a community per se, but it originated from Linux and perhaps that was not an accident either.)
> FreeBSD pioneered the practical implementation of what we now call containers.
Ok great. Many modern programming languages learned from older languages; many of these older languages are dead now. You need to keep on innovating. Why is BSD so dead set on the past?
> FreeBSD reached that third stage in 2000. Linux wouldn't get there until 2008 with LXC.
Dumdedum ... it kind of sounds as if the FreeBSD guys are sad that Linux went on to dominate. It reminds me of NetBSD aka "we work on every toaster in the world". Then suddenly on a mailing list many years ago "wait a moment ... Linux now works on more toasters than we do". The BSDs don't seem to understand how momentum can be dominating.
> Technical superiority doesn't win ecosystem wars. Linux won through a combination of fast decisions, the viral GPL licence, and strong enterprise backing from Red Hat and IBM. Then Google, Facebook, and Amazon happened — hungry for datacenters, developing tools to manage growing infrastructure at scale. They set the direction for the entire industry.
Ok that flat out is incorrect. First - GPL worked well for the linux kernel, that is true. But the ecosystem includes many BSD-licences programs too, on Linux. So that explanation fails already here. LLVM has Apache License 2.0 which I kind of feel is a mix between GPL and BSD (not quite true but this is how I remember it).
Then the claim is Linux won because of Red Hat. I actually find Red Hat annoying and I am glad to not depend on it. Linux is way bigger than Red Hat. IBM? I don't see what IBM did for Linux really. So that explanation also does not work.
Google, Facebook, and Amazon - well, they profited from Linux. They didn't really ENABLE Linux. They would not have used Linux if Linux would have been useless. So that part came afterwards.
So none of those explanations really work well here.
> Linux rapidly went from "the free OS for people who can't afford commercial licences" to "the only acceptable OS for servers".
That is true but not for the claims made, e. g. "because of Google". The more important question is: why did the BSDs fail?
> To solve the distribution and isolation problem, Linux engineers built a set of kernel primitives (namespaces, cgroups, seccomp) and then, in a very Linux fashion, built an entire ecosystem of abstractions on top to “simplify” things
> Somehow we ended up with an overengineered mess of leaky abstractions for cloud-based, vendor-locked infrastructure.
Wait a moment - he cites Docker. That's owned by a private company. What does this have to do with Linux? If company xyz does something based on FreeBSD, we would then say company xyz is responsible for FreeBSD failing or not failing? How does that work?
> And this complexity has quietly reshaped how the industry thinks about deploying software. Today, if you want to run an application in a larger system, the implicit assumption is that you containerise it with Docker and orchestrate it with Kubernetes.
Personally I find all this abstraction crap. With all their failures, though, things such as docker kind of present a "download this one file, then it will work fine". And that is kind of true. I saw that in in-campus use for life science faculty clusters and what not. It simplifies things for the admin there. People give a similar rationale for systemd. Personally I don't think systemd should exist, but there are people who benefit from it - that simply is a factual statement.
All in all this is a very strange point of view from FreeBSD folks. At the least the NetBSD folks back then on the mailing list acknowledged the situation and then tried to find alternative strategies and in some ways succeeded (although I am not sure whether NetBSD right now runs on more toasters than Linux does - anyone has updated statistics for that?).
>>I think 500 out of 500 top supercomputers running Linux kind of show which philosophy is better.
Or is it because it's what they're used to. I saw this argument elsewhere where the respondent went on to show that the users were Linux specialists and that's why Linux was used.
FreeBSD doesn't have support for CUDA, many third party network fabrics and accelerators, Lustre, and other high performance storage solutions. It makes FreeBSD for frontier machines a non-starter. FreeBSD adoption for AI trainer/inference clusters has the same problems.
Getting the same thing, "Failed to verify your browser. Code 11". Some noise about WebGL in the browser console, getExtension() invoked on a null reference. LibreWolf on Linux + resist fingerprinting.
Maybe opting for a better-written WAF could boost the reach?
> Technical superiority doesn't win ecosystem wars. Linux won through a combination of fast decisions, the viral GPL licence, and strong enterprise backing from Red Hat and IBM. Then Google, Facebook, and Amazon happened — hungry for datacenters, developing tools to manage growing infrastructure at scale. They set the direction for the entire industry.
In the mid 1990's the hardware driver support on Linux was much broader.
Copy / paste of my comment from last year about FreeBSD
I installed Linux in fall 1994. I looked at Free/NetBSD but when I went on some of the Usenet BSD forums they basically insulted me saying that my brand new $3,500 PC wasn't good enough.
The main thing was this IDE interface that had a bug. Linux got a workaround within days or weeks.
https://en.wikipedia.org/wiki/CMD640
The BSD people told me that I should buy a SCSI card, SCSI hard drive, SCSI CD-ROM. I was a sophomore in college and I saved every penny to spend $2K on that PC and my parents paid the rest. I didn't have any money for that.
The sound card was another issue.
I remember software based "WinModems" but Linux had drivers for some of these. Same for software based "Win Printers"
When I finally did graduate and had money for SCSI stuff I tried FreeBSD around 1998 and it just seemed like another Unix. I used Solaris, HP-UX, AIX, Ultrix, IRIX. FreeBSD was perfectly fine but it didn't do anything I needed that Linux didn't already do.
> FreeBSD was perfectly fine but it didn't do anything I needed that Linux didn't already do.
That's pretty much it. A lot of the people I see using a BSD these days do so because they always have and they prefer what they know, which is fine, or they just want to be contrarian.
Realistically, aside from edge cases in hardware support, you can do anything you want on any modern *nix. There's not even as much of a difference between distros as people claim. All the "I want an OS that gets out of my way" and similar reasons apply to most modern well-maintained distros these days. It's more personality and familiarity than anything objective.
I went from Slackware in 1994 to Red Hat in 1998 to Fedora when they split into Fedora and RH Enterprise. Every 2 or 3 years I will install a different distro in a VM and see "Okay, now I see what it's about." But I have no interest in switching as long as Fedora does everything I need. I don't really understand the people that distro hop. I just assume they are really young and I have work to do and a family to take care of.
I get that. I stopped using Slack around...not sure, maybe 2007 or so when I tried to do my normal minimal setup and mplayer wouldn't run without Samba installed, and the community was hostile to anyone who didn't do the recommended full install. I never wanted it a a feature complete desktop but that's the market they tried to focus on.
Took me a while to settle on Alpine after trying Arch and Void, but I can't imagine why I would ever leave unless they change something drastic.
I mean… at this point Linux is nothing like the 90s. It’s just big corporate Unix. Every change is in the interest of massive cloud providers.
> FreeBSD was perfectly fine but it didn't do anything I needed that Linux didn't already do.
I broadly agree, even as a FreeBSD fan myself; things have converged a lot over the decades. But still, I generally feel that while you can get the same work done in both, FreeBSD does things better (and/or cleaner, more elegant, etc) in many cases.
The overall feeling of system cohesion makes me happier to use it, from small things like Ctrl-T producing meaningful output for all the base OS tools, to larger and more amorphous things like having greater confidence core systems won't change too quickly over time (eg: FreeBSD's relatively stable sound support, versus Linux's alsa/pulse/pipewire/..., similar for event APIs, and more).
Though I totally feel your pain about latest-and-greatest hardware driver support. Has gotten better since the '90s, but that gap will probably always be there due to the different development philosophies.
I hope FreeBSD never gets too "Linux-y"; it occupies it's own nice spot in the spectrum of available options.
> they basically insulted me saying that my brand new $3,500 PC wasn't good enough.
Big chuckle there, so good. Hey, at least they had a sense of humour.
But I agree the hardware support could be much better even to this day.
My feeling at the time was that BSD developers and many users were at least 30 years old or much older and had professional jobs and money.
The Linux community felt like college students with no job and not much money. That included Linus Torvalds himself who developed the kernel while in college and wasn't rich. DEC basically gave Linus an Alpha to get him to port the kernel to it.
Nice article!
> To solve the distribution and isolation problem, Linux engineers built a set of kernel primitives (namespaces, cgroups, seccomp) and then, in a very Linux fashion, built an entire ecosystem of abstractions on top to “simplify” things: [...] Somehow we ended up with an overengineered mess of leaky abstractions
Not sure I like the value judgement here. I think it's more of a consequence of Linux' success. I am convinced that if it was reversed (Linux was niche and *BSD the norm), then a ton of abstractions would come, and the average user would "use an overengineered mess" because they don't know better (or don't care or don't have a need to care).
Not that I like it when people ship their binary in a 6G docker image. But I don't think it's fair to put that on "those Linux engineers".
I don't agree with that. FreeBSD has more of an engineering than a hacking mentality and it shows in the various architectural choices.
And containers really are a VM-light, so you might as well use the real thing, in fact, VMWare for a long time thought that their images would be a container like thing and many larger installations used them as such.
I don't think it's necessarily true, compare the BSD utils to the GNU utils and the style difference is very visible.
On the other hand, I don't think the comparison between jails and docker is fair. What made Docker popular is the reusability of the containers, certainty not the sandboxing which in the early days was very leaky.
Indeed, that Docker is functionally a cross-distro rolling release package manager, configuration standard, and service supervisor[1] is the appeal to me. Any isolation it achieves is necessary for that to all work reliably, but is not why I use it.
Inability to find a service I want to run on Github and 95+% of the time to be able to configure it and have it running and fully managed with usually just a one-liner shell script like 10 minutes later just by finding an existing docker image is the thing I’d lose with jails. That’s all of the value of docker to me personally. Jails could be a building block toward that, but last I checked there’s no deep and up-to-date library of “packages” I can reach for, using jails, which makes it pretty much useless to me.
1: I have like eight or nine services running on my home Debian system, they all auto-restart and come back up on reboot, and I’ve not had to touch Systemd once on that machine.
> compare the BSD utils to the GNU utils and the style difference is very visible.
Well, what style difference exactly? GNU utils tend to be more verbose. Other than that, what is the difference in style?
I do not know which is the difference, but you really feel a difference.
It might be of homogeneity, i.e. the FreeBSD tools behave in a consistent way, while there are significant differences between the Linux tools, depending on which were the opinions of their particular authors about how the traditional UNIX tools should be changed.
For instance, at some point in time, long ago, in Linux the traditional "ifconfig" and a few related commands have been replaced by "ip", for managing networking.
The Linux "ifconfig" needed an upgrade, as it could do only a small fraction of what the FreeBSD "ifconfig" could do. Nevertheless, until today, decades later, I have been unable to stop hating the Linux "ip".
I cannot say why, because in other cases when some command-line or GUI utility that I had used for many years was replaced by an alternative I instantly recognized that the new UI was better and I never wanted to use the old UI again.
So while both FreeBSD and Linux have started with the same traditional UNIX utilities, they have evolved divergently and now they frequently feel quite differently, in the sense that the various options in commands or in configuration files may match your expectations only when taking into account the identity of the OS. Overall FreeBSD has been more conservative, but there are also cases when it has made bigger changes, but such changes seem more carefully planned and less haphazard than in the Linux world.
what do you mean by reusability?
For example, you can build a Python image, and reuse it on every Python apps you have.
And for the whole world, too. I don't need to build my own local stripped down version of Alpine Linux with python, somebody's already dike that for me.
I don't like that aspect of OCI containers. You shouldn't be running or building on top of random images made by unknowns.
Convenience beats security every time.
I frequently see freeBSD jails as a highlighted feature, lauding their simplicity and ease of use. While I do admire them, there are benefits to the container approach used commonly on linux. (and maybe soon freebsd will better support OCI).
First it's important to clarify "containers" are not an abstraction in the linux kernel. Containers are really an illusion achieved by use of a combination of user/pid/networking namespaces, bind mounts, and process isolation primitives through a userspace application(s) (podman/docker + a container runtime).
OCI container tooling is much easier to use, and follows the "cattle not pets" philosophy, and when you're deploying on multiple systems, and want easy updates, reproducibility, and mature tooling, you use OCI containers, not LXC or freebsd jails. FreeBSD jails can't hold a candle to the ease of use and developer experience OCI tooling offers.
> To solve the distribution and isolation problem, Linux engineers built a set of kernel primitives (namespaces, cgroups, seccomp) and then, in a very Linux fashion, built an entire ecosystem of abstractions on top to “simplify” things.
This was an intentional design decision, and not a bad one! cgroups, namespaces, and seccomp are used extensively outside of the container abstraction. (See flatpak, systemd resource slices, firejail). By not tieing process isolation to the container abstraction, we can let non-container applications benefit from them. We also get a wide breadth of container runtime choices.
Jails have been around a long time in comparison
I still see FreeBSD as being great for things like networking devices and storage controllers. You can apply a lot of the "cattle vs pets" design one level above that using VMs and orchestration tools.
> lauding their simplicity and ease of use
Spawning a linux container is much simpler and faster than spawning a freebsd jail.
I don’t know why i keep hearing about jails being better, they clearly aren’t.
If you don't want to use the base system (which docker is NOT the base system on Linux) then Bastille offers a pretty much identical workflow to docker, but built on FreeBSD jails: https://github.com/BastilleBSD/bastille
> I don’t know why i keep hearing about jails being better
Jails have a significantly better track record in terms of security.
I can delegate a ZFS dataset to a jail to let the jail manage it.
Do Linux containers have an equivalent to VNET jails yet? With VNET jails I can give the jail its own whole networking stack, so they can run their own firewall and dhcp their own address and everything.
You've been able to setup separate firewalls, network interfaces, IP addresses, etc. for probably 20 years using network namespaces. How do you think container networking is implemented? But you can also use it through other tools; for example, I use firejail to isolate a couple of proprietary desktop applications such that they cannot contact anything on my desktop (or network in general) except the internet gateway.
Is there a docker-compose analogue in Bastille? I like being able to spin up an isolated local copy of my infrastructure, run integration tests, and then tear it all down automatically. I'd like to be able to do a similar thing with jails. I wonder if there's a straightforward way to achieve something similar with VNET jails?
Not that I'm aware of. FreeBSD did recently gain support for OCI containers and therefore has podman. I see podman-compose is in the ports tree, but I haven't tried it myself.
Sorry what? It's a 5 line configuration file to create a FreeBSD jail.
The jails vs containers framing is interesting but I think it misses why Docker actually won. It wasn't the isolation tech. It was the ecosystem: Dockerfiles as executable documentation, a public registry, and compose for local dev. You could pull an image and have something running in 30 seconds without understanding anything about cgroups or namespaces.
FreeBSD jails were technically solid years before Docker existed, but the onboarding story was rough. You needed to understand the FreeBSD base system first. Docker let you skip all of that.
That said, I've been seeing more people question the container stack complexity recently. Especially for smaller deployments where a jail or even a plain VM with good config management would be simpler and more debuggable. The pendulum might be swinging back a bit for certain use cases.
Jails were never going to 'win' because they're only on an OS with 0.1% marketshare.
But it's not a competition. FreeBSD does its thing and Linux does another. That's why I use FreeBSD.
What is your use case for BSD?
I love several things about it:
- Stable OS coupled with rolling packages. I am on the previous FreeBSD version (14.3-RELEASE, while 15 is out) but I have the very latest KDE.
- A ports collection where you can recompile packages whenever you're not happy with the default settings. Strict separation between packages and core OS. Everything that is from packages is in /usr/local (and this separation is also what makes the above point possible).
- ZFS on root as first-class citizen. Really great. It has some really nice side tooling too like sanoid/syncoid and bectl (the latter is part of the core OS even).
- jails for isolation (I don't really use it like docker for portability and trying things out)
- Clear documentation because there are no different distros. Very good handbook. I like the rc.conf idea.
- Common sense mentality, not constantly trying to reinvent the wheel. I don't have to learn a new init system and I can still use ifconfig. Things just work without constantly being poked around.
- Not much corporate messing around with the OS. Most of the linux patches come from big tech now and are often related to their goals (e.g. cloud compatibility). I don't care about those things and I prefer something developed by and for users, not corporate suits. No corporates trying to push their IP onto the users (e.g. canonical with their Mir, snaps etc)
- Not the same thing as everyone else has. I'm not a team player, I hate going with the flow. I accept that sometimes comes with stuff to figure out or work around.
I think that's about it.
ZFS boot+root on Linux is amazing as well. It's kind of sad to see Linux Mint has moved away from supporting this in their installer, but it probably could still be done manually I guess. After upgrade, if something goes wrong? zfs rollback both to a snapshot made just before the upgrade and reboot.
I don't use it full time, only in a VM, but all these things sound positive to me.
*everything. I've really been using it since 4.x. Imagine this: being able to upgrade a system in-place with freebsd-update from minor to major to minor version without everything breaking or having to say a prayer before. And that's just one thing I love about it. Clear separation of userland (/usr/local/etc), rock-solid stability in networking, zfs on root.
I had to do 'bonded' interfaces on Debian the other day. It's what, 5 different config files depending on which 'network manager' you use. In FreeBSD it's 5 lines in /etc/rc.conf and you're done.
And don't even get me started on betting which distribution (ahem CentOS) will go away next.
Centos didn’t go away. It changed. Rocky (et. al.) took the old centos role, and I see this as a win/win for everybody.
Ubuntu is the disaster Linux distro, I won’t touch Ubuntu if I have any other option.
I actually laughed out loud. Try upgrading CentOS to Rocky vs FreeBSD 11 to 15 ( that's FOUR major versions from 2017 I think ), and tell me again how good it is.
In LTS environments where I need to upgrade OS's, FreeBSD is a no-brainer.
> I actually laughed out loud. Try upgrading CentOS to Rocky vs FreeBSD 11 to 15 ( that's FOUR major versions from 2017 I think ), and tell me again how good it is.
I laughed out loud, there is no in-place upgrade mechanism for that in those distros and never has been, that is the nature of those distros. They release patch/security updates until they go EOL, which is measured in units closer to decades than years.
I don’t have a problem with BSDs. That’s cool you like upgrading in place.
The best and most laugh-inducing part of your whole point is that centos now not only allows you to do in-place upgrades, that’s the whole fucking point.
So then what's the point of mentioning Rocky as CentOS's successor ? In what way is it 'succeeding' ? That you can do a fresh install of Rocky ? And those stuck on CentOS can't upgrade ? Really useful those decades of support if your distro goes belly up
> the container stack complexity
I'm using either Docker Compose or Docker Swarm without Kubernetes, and there's not that much of it, to be honest. My "ingress" is just an Apache2 container that's bound to 80/443 and my storage is either volumes or bind mounts, with no need for more complexity there.
> The jails vs containers framing is interesting but I think it misses why Docker actually won. It wasn't the isolation tech. It was the ecosystem: Dockerfiles as executable documentation, a public registry, and compose for local dev. You could pull an image and have something running in 30 seconds without understanding anything about cgroups or namespaces.
So where's Jailsfiles? Where's Jail Hub (maybe naming needs a bit of work)? Where's Jail Desktop or Jail Compose or Jail Swarm or Jailbernetes?
It feels like either the people behind the various BSDs don't care much for what allowed Docker to win, or they're unable to compete with it, which is a shame, because it'd probably be somewhere between a single and double digit percent userbase growth if they decided to do it and got it right. They already have some of the foundational tech, so why not the UX and the rest of it?
I think Jails started as a tool of it's time, it's about the same thing as virtualization in making isolated systems when dependencies start to diverge, but aimed at the issues of sysadmins that had to manage their own systems, not a quick developer experience.
Even if "jailsfiles" were created the ecosystem would need to start from scratch and sometimes it feels like people in the FreeBSD ecosystem have a hard enough time keeping ports somewhat up to date, let alone create something new.
Luckily Podman seems to support FreeBSD these days for docker images, but the Linux emualation might be a bit of a blocker so not a 100% solution.
> I'm using either Docker Compose or Docker Swarm without Kubernetes, and there's not that much of it, to be honest.
On the outside. But that's a lot of complexity hidden from view there, easily a couple of million lines of code on top of the code that you wrote.
I never used this, but noticed it in some docs back when I was using Nomad and thought it was an intriguing idea: https://github.com/cneira/jail-task-driver
Docker's client/server design also allowed for things like Docker Desktop, which made the integration seamless with non-linux systems. Jails have nothing like that, so the only system that will ever run jails is FreeBSD. Also, I'm not up to speed enough to know, but do jails even have a concept of container images?
It’s just files on the filesystem. So tar for imaging?
Plus a script to unpack the tarball somewhere and launch some entry point in a jail. Not conceptually hard, but the OCI spec has a bit more to it than that, and now we're into "write dropbox with rsync" territory...
I did some looking around, and I see that ocijail is a thing, so that's probably what I was looking for.
(edited, sorry, I didn't see your reply)
What do you mean”launch an entry point”? The rc script would naturally be included.
I don't think article misses it, it's exactly the point it makes
> Jails solve the isolation problem beautifully, but they don't have a native answer to shipping. That gap is real, and it's one of the main reasons the ecosystem around jails feels underdeveloped compared to Docker's world.
The link literally uses the term ecosystem. Several times actually.
You can also run Linux containers on FreeBSD
https://youtu.be/HV-wUUzRCMo
I've tried this ... I've haven't got much mileage on this, sadly.
Many Linux syscalls are unemulated and things like /proc/<pid>/fd/NN etc are not "magic symlinks" like on Linux so execve on them fails (i.e there is rudimentary /proc support, it's not full fleshed out).
TL;DR Linux containers on FreeBSD via the podman + linuxulator feel half baked.
For example, try using the alpine container... `apk upgrade` will fail due to the /proc issue discussed above. Try using the Fedora container `dnf upgrade` will fail due to some seccomp issue.
The future of containers on FreeBSD is FreeBSD OCI containers, not (emulated) Linux containers. As an aside, podman on FreeBSD requires sudo which kinda defeats the concept but hopefully this will be fixed in the future.
Maybe FreeBSD doesn't want a jails "ecosystem"?
> You could pull an image and have something running in 30 seconds without understanding anything
Fixed that for you ;)
I ran a whole company on top of FreeBSD back in the day (2005 ish). It was great, and ran all my personal pcs the same way (hell, refusing to install windows to try out this bitcoin idea is even now a good idea).
But somehow Linux still took over my personal and professional life.
Going back seems nice but there need to be a compelling reason -docker is fine, the costs don’t add up any more. I do t have a real logical argument beyond that.
In the early years after 2000, FreeBSD 4 had a much better performance and reliability in any networking or storage applications in comparison with the contemporaneous Linux and Windows XP/Windows 2000.
However, in 2003 Intel introduced CPUs with SMT and in 2005 AMD introduced multi-core CPUs.
These multi-threaded and/or multi-core CPUs quickly replaced the single-threaded CPUs, especially in servers, where the FreeBSD stronghold was.
FreeBSD 4 could not handle multiple threads. In the following years Linux and Windows have been developed immediately to take advantage of multiple threads and cores, while FreeBSD has required many years for this, a time during it has become much less used than before, because new users were choosing Linux and some of the old users were also switching to Linux for their new computers that were not supported by FreeBSD.
Eventually FreeBSD has become decent again from the PoV of performance, but it has never been again in a top position and it lacks native device drivers for many of the hardware devices that are supported by Linux, due to much fewer developers able to do the necessary reverse engineering work or the porting work for the case when some company provides Linux device drivers for their hardware.
For the last 3 decades, I have been using continuously both FreeBSD and Linux. I use Linux on my desktop PCs and laptops, and in some computational servers where I need software support not available for FreeBSD, e.g. NVIDIA CUDA (NVIDIA provides FreeBSD device drivers for graphic applications, but not CUDA). I continue to use FreeBSD for many servers that implement various kinds of networking or storage functions, due to exceptional reliability and simplicity of management.
The FreeBSD threading was perhaps behind but in general, but the big things in Linux VS FreeBSD was always the 4.3 licensing lawsuits that gave Linux a momentum that BSD never caught up with.
The real difference during that early 00s was that momentum bought 2 things that made FreeBSD a worse choice (and made even more people end up using Linux):
1: "commercial" support for Linux, firstly hardware like you mentioned, but in the way that you could buy a server with some Linux variant installed and you knew that it'd run, unless you're an CTO you're probably not risking even trying out FreeBSD on a fresh machine if time isn't abundant.
Also software like Java servers comes to mind, came with binaries or was otherwise easy to get running on Linux, and even with FreeBSD's Linux layer VM's things like JVM and CLR often relied on subtle details that makes it incompatible with the Linux layer (tried running .NET a year or two ago, ran into random crashes).
2: a lot of "fresh" Linux developers had a heavy "works on my machine" mentality, being reliant on Linux semantics, paths or libraries in makefiles (or dependencies on things like systemd)
Sure there is often upstream patches (eventually) or patches in FreeBSD ports, those last are good for stable systems, but a PITA in the long run since stuff doesn't get upstreamed properly and you're often stuck when there is a major release and need to figure out how to patch a new version yourself.
Yeah, I have a similar situation; FreeBSD is a great operating system, but the sheer amount of investment in Linux makes all the warts semi-tolerable.
I'm sure some people have a sunk-cost feeling with Linux and will get defensive of this, but ironically this was exactly the argument I had heard 20 years ago - and I was defensive about it myself then.. This has only become more true though.
It's really hard to argue against Linux when even architecturally poor decisions are papered over by sheer force of will and investment; so in a day-to-day context Linux is often the happy path even though the UX of FreeBSD is more consistent over time.
Never understood why satoshi was a prime windows user.
I know this comment is effectively a side tangent on a side tangent. but that was always the strangest thing to me as well. I remember in 2012 when I was debating fiddling around with Bitcoin. that was one of the things that turned me off. I was sure that there was no way something as brilliant as this was supposed to be was developed by windows user.
Which surely says something about all these ideological purity tests
Windows developers (like sysadmins) are of two kinds in my experience.
People who don't understand shit about how the system behaves and are comfortable with that. "I install a package, I hit the button, it works"
.. and
People who understand very deeply how computers work, and genuinely enjoy features of the NT Kernel, like IOCP and the performance counters they offer to userland.
What's weird to me is that the competence is bimodal; you're either in the first camp or the second. With Linux (+BSD/Solaris etc;) it's a lot more of a spectrum.
I've never understood exactly why this is, but it's consistent. There's no "middle-good" Windows developer.
The (install package, press button, it works) is great when you just want a boring OS since the interest is elsewhere rather than an itch on making the machine as perfect extension of onself.
The machine and installation is just fungible.
I think I've had Linux as a primary OS 2 times, FreeBSD once and osX once, what's pulled me back has been software and fiddling.
I'm on the verge of giving Linux or osX another shot though, some friends has claimed that fiddling is virtually gone on Linux these days and Wine also seems more than capable now to handle the software that bought me back.
But also, much of the software is available outside of Windows today.
Probably bc, Windows users live in walled knowledge domains that tend to reinforce levels of competence (or lack of competence).
Gamers tend to be somewhere in the middle though.
Unix is easier to understand than the NT mess and everything it's in the open and documented, so you can achieve a good level of knowledge in the middle. OTOH in order to understand NT deeply you must be a reverse engineer. Also, on the other side, crazy experts under Wine (both ways, Unix and NT) OpenBSD and 9front do exist on par of these NT wizards. It just happen with Unix/9f you climb an almost flat slope (more in the second) due to the crazy simple design, while with NT the knowledge it's damn expensive to earn.
With 9front you OFC need expertise on par of NT but without far less efforth. The books (9intro), the papers, CSP for concurrency... it's all there, there's no magic, you don't need ollyDBG or an NT object explorer to understand OLE and COM for instance.
RE 9front? Maybe on issues while debugging, because the rest it's at /sys/src, and if something happens you just point Acid under Acme to go straight to the offending source line. The man pages cover everything. Drivers are 200x smaller and more understandable than both NT and Unix. Meanwhile to do that under NT you must almost be able to design an ISA by yourself and some trivial compiler/interpreter/OS for it, because there's no open code for anything. And no, Wine is not a reference, but a reimplementation.
That's kinda true for older/integrated parts of Windows, lots and lots of functionality that people have come to rely on over the years, but also huge black-boxes that you need to not be intimidated at probing into to solve weird issues (that often becomes understandable if you have enough experience as a developer to interpret what the API surface tells about the possible internal implementation).
I’m always going to like articles introducing people to FreeBSD.
> FreeBSD reached that third stage in 2000. Linux wouldn't get there until 2008 with LXC.
OpenVZ and Linux vserver are older than LXC and were commonly used, though they required a patched kernel.
Yeah, both VServer and Virtuozzo were roughly contemporaneous with the initial Xen release (2001-ish?), and only a few months behind jails.
But Virtuozzo was hampered by being non-free - OpenVZ wasn't open sourced until 2-3 years later, by which time the damage had been done (but, of course, Xen headed in the opposite direction at roughly the same time!)
And Linux-VServer was held back by being focussed so directly at virtual hosting providers - it positioned itself against fcgi-suexec, fcgid, and php-fpm (and was much more unwieldy than any of them) rather than jails or VZ.
Both were more or less ignored until the late 2000s, by which time LXC had taken a lot of mindshare - allowing the "FreeBSD was years ahead with jails" meme to take root.
Is there any technical writeup which explains how the isolation exactly works, on containers and VMs? I have always heard the high level arguments of weak isolation, same kernel, etc but never the implementation details.
Long emotional rant ahead, you have been warned. The poorly thought out adoption of parts of systemd, and in particular systemd-oomd, are making me yearn for FreeBSD.
For all my computing career, I'd use Unix-alike because they let me develop software by having an idea, write some code, let it run in the terminal, chain things together, and see where it failed, and iterate. Terminals let this happen much faster than clicky GUI software, and contain a log of what was happening in a terminal pane (or gnus screen or tmux pane, because usually this is happening on big servers and compute clusters rather than my terminal/laptop)
I could launch a bunch of panes, have several lines of investigation going, and come back to it a day or week later when I had time because the terminal kept a log of everything that happened.
And a couple years ago I started noticing that things I thought I had launched would start disappearing. At first I thought I had started accidentally mispressing a key and killing a tmux pane rather than disconnecting.
But no! When I finally went back to a in-person Linux workstation and saw it happen to the entire terminal window, I knew something major had changed. It turns out that something called systemd-oomd was added to a bunch of distributions that now kills only entire cgroups of processes at a time, rather than a single process offender.
So now if you want to run processes and isolate the kill zone of a process, you have to wrap every freaking subprocess in an entire systemd-run wrapper or docker wrapper. And systemd-run won't work from many contexts, such as inside a Jupyter kernel.
Major breaking changes on fundamental system behavior are a huge problem these days. It's one thing to let the OS kill processes more when there's a memory issue, fine, great, go ahead. But why kill all the lightweight processes that could give feedback to a user?! And why force non-portable process launching semantics, that aren't even consistent across the entire system?!? So infuriating.
Anyone looking to use jails might find BastileBSD helpful. It's a nice and modern jail manager.
I was looking at TrueNAS CORE to see if it was a viable way to bsd-jail Linux containers. I'm really only doing this to get some protection from supply chain attacks given I'm fairly promiscuous at git-clone-and-run-a-build. Before that I was aiming for the same with Bastille and had got to the give up stage because it felt too fiddly to set up. This was a year ago. Maybe its better now
A two-server networking setup with VNET Jails:
https://ericfortis.com/blog/freebsd-jails-network-setup
>but they don't have a native answer to shipping
I am not quite sure what this means. I had a jail a few years ago and I remember there was a utility to "back" the jail up so you could put it on another system. Are there constraints with that utility. It seemed to work, maybe I am forgetting something ?
In any case I still think Jails are much better than the things Linux has. To me, it is creating a jail that is more difficult. There were ports that made it easier, I used one of them, but that port was abandoned at some point. I think it was "ezjail".
I switched my startup’s whole infra to FreeBSD a couple months ago. Found a use after free bug that Linux’s memory management was just fine with in Gnome XSLT lib that FreeBSD properly refused. Other than that smooth sailing, jails work great.
After IBM destroyed CentOS, all the Xorg politics nonsense, the list goes on with Linux, not interested. I just want something quiet and boring and stable and correctly designed. NetBSD would be my first choice but they don’t get the $ they need for drivers.
Done the same since 2018 circa, never looked back.
For a while even used it on the desktop, but was too much trouble due to specific tools we need that weren't supported properly. so we're using Linux on the desktop.
FreeBSD is stable, lightweight, gets out of the way, and without drama.
You don’t need to follow the news cycle to use an operating system.
I do follow the news cycle and if I’m hearing about a software package in it, something is wrong with the people making the software and I don’t trust them. Software is an engineering discussion or at least it’s supposed to be. Here’s my community guidelines: everyone be nice and respectful engage in good faith and focus on the math. Being social is fine so long that it doesn’t become a diversion from the engineering discussion. We’re talking about code not a philosophical treatise. There are civil ways to settle disagreements. I’m so sick to death of the politics.
It sounds like software projects are built by humans. Nothing wrong with that.
Unless we’re assuming here that the BSD community is free from that.
The engineers I’ve worked with my whole career had no problem getting the work done without getting into giant political debates. If your politics comes before the engineering I don’t want anything to do with it.
Can you give any examples of the type of politics that ruined XOrg or Linux?
> all the Xorg politics nonsense
Uh... Xorg is packaged by FreeBSD too...
Really the whole theme that (from the article) "FreeBSD ships as a complete, coherent OS" is belied by this kind of nonsense. No, it's not. Or, sure, it is, but in exactly the same way that Debian or whatever is. It's a big soup of some local software and a huge ton of upstream dependencies curated for shipment together. Just like a Linux distro.
And, obviously, almost all those upstream dependences are exactly the same. Yet somehow the BSD folks think there's some magic to the ports stuff that the Linux folks don't understand. Well, there isn't. And honestly to the extent there's a delta in packaging sophistication, the Linux folks tend to be ahead (c.f. Nix, for example).
The key thing is that on freebsd you do not risk bricking your system by installing a port. Even though this guarantee has become less true with PkgBase
> The key thing is that on freebsd you do not risk bricking your system by installing a port
What specifically are you trying to cite here? Which package can I install on Debian or Fedora or whatever that "bricks the system"? Genuinely curious to know.
I was referring to the need to be careful to not modify/update packages also used by the base system. Since all packages are treated the same on Linux, you often can't tell which package can put you in trouble if you update it along with the dependencies it drags with it.
This kind of problem happens frequently when users add repositories such as Packman on Linux providing dependencies versions different from the ones used by the base system of the distro.
Experienced people know how to avoid these mistakes, but this whole class of problem does not exist on FreeBSD.
> Since all packages are treated the same on Linux
This is no longer the case in "immutable" distros such as Bluefin/Aurora, which uses ostree for the "base" distro, while most other user packages are installed with homebrew. Nix and Guix solve it in a very different way. Then there's flatpak and snap.
A lot of poor *BSD advocacy likes to deride Linux for its diversity one moment, then switch to treating it as a monolith when it's convenient. It's a minority of the users for sure, but they naturally make an outsized share of the noise.
> a huge ton of upstream dependencies
I think you missed the point in my original comment. I explained I moved my platform with all dependencies and had 1 bug which was actually a silent bug in Linux.
In other words, it works. Your particular stack might have a different snag profile but if I can move my giant complex app there, yours is worth a shot.
FreeBSD is more complete than you make out. They also have hard working ports maintainers.
> In other words, it works.
Well, sure, but that's a ridiculous double standard. You're making the claim (or implying it, at least) that FreeBSD is fundamentally superior because it's a unified piece of software shipped as a holistic piece of artifice or whatever. And that by inferrence it's unlike all that kludgey linux stuff that you can't trust because of politics or whatever.
But your evidence that it's actually superior? "it works". Well, gosh.
You'll tar the competition with all sorts of ambiguous smears, but all you ask from your favorite is... that you got your app to work?
Is this fair?
Linux is to *BSD as
VHS was to Betamax.
At one time this was an interesting comparison... but now Linux has gotten so much development that even if FreeBSD was Betamax and Linux VHS (in the past)... I would say that Linux is now DVD ... and FreeBSD still remains betamax.
Don't get me wrong, FreeBSD is simple, elegant, consistent and well manicured. It seems to have picked up some pace again. I'm rooting for it.
Indeed, there is no shame in being a dirt-simple system that Just Works.
Likely is has a good place at the low end.
In enterprise mode, you want something like an AWS to hide the pain of those large-scale details that Linux is bringing.
> FreeBSD is worth a brief aside here, because it differs from Linux in a fundamental way. Linux is a kernel. What most people call "Linux" is actually that kernel combined with a GNU userland, a package ecosystem, and a set of choices that vary from distro to distro — Ubuntu, Fedora, and Arch are all running the same kernel but are meaningfully different systems underneath.
It is not incorrect but ... do people really care about that distinction?
Because in most situations I know of, when people refer to Linux, they almost never refer to the linux kernel. They refer to the whole operating system stack, which is typically put down via a distribution. So, Fedora, Gentoo, Arch, and so forth, are all "kind of" Linux. Barely anyone refers to the linux kernel if you look at all the discussions on the world wide web.
> FreeBSD ships as a complete, coherent OS
The BSDs often promote that aka "Linux is chaos, we are coherent and consistent operating system following intelligent design". Well ... this is the rise of worse is better, repeated: https://dreamsongs.com/WorseIsBetter.html
It is a great analogy that works on so many levels. Broken down to Linux versus the BSDs, I think 500 out of 500 top supercomputers running Linux kind of show which philosophy is better. The one that works better. That does not mean the BSDs are useless, but I am getting tired of the promo used by the BSD as "we are order, Linux is chaos". I compare this more to Lego building blocks. With Linux there is a stronger focus on having building blocks available. You can build up things. You have projects such as LFS/BLFS (Linux from scratch). The BSDs do not have something comparable. Which operating system is the better tinker OS? Which community created git? (Ok ok that was Linus so not really a community per se, but it originated from Linux and perhaps that was not an accident either.)
> FreeBSD pioneered the practical implementation of what we now call containers.
Ok great. Many modern programming languages learned from older languages; many of these older languages are dead now. You need to keep on innovating. Why is BSD so dead set on the past?
> FreeBSD reached that third stage in 2000. Linux wouldn't get there until 2008 with LXC.
Dumdedum ... it kind of sounds as if the FreeBSD guys are sad that Linux went on to dominate. It reminds me of NetBSD aka "we work on every toaster in the world". Then suddenly on a mailing list many years ago "wait a moment ... Linux now works on more toasters than we do". The BSDs don't seem to understand how momentum can be dominating.
> Technical superiority doesn't win ecosystem wars. Linux won through a combination of fast decisions, the viral GPL licence, and strong enterprise backing from Red Hat and IBM. Then Google, Facebook, and Amazon happened — hungry for datacenters, developing tools to manage growing infrastructure at scale. They set the direction for the entire industry.
Ok that flat out is incorrect. First - GPL worked well for the linux kernel, that is true. But the ecosystem includes many BSD-licences programs too, on Linux. So that explanation fails already here. LLVM has Apache License 2.0 which I kind of feel is a mix between GPL and BSD (not quite true but this is how I remember it).
Then the claim is Linux won because of Red Hat. I actually find Red Hat annoying and I am glad to not depend on it. Linux is way bigger than Red Hat. IBM? I don't see what IBM did for Linux really. So that explanation also does not work.
Google, Facebook, and Amazon - well, they profited from Linux. They didn't really ENABLE Linux. They would not have used Linux if Linux would have been useless. So that part came afterwards.
So none of those explanations really work well here.
> Linux rapidly went from "the free OS for people who can't afford commercial licences" to "the only acceptable OS for servers".
That is true but not for the claims made, e. g. "because of Google". The more important question is: why did the BSDs fail?
> To solve the distribution and isolation problem, Linux engineers built a set of kernel primitives (namespaces, cgroups, seccomp) and then, in a very Linux fashion, built an entire ecosystem of abstractions on top to “simplify” things
No, that is also incorrect. cgroups are also very different to seccomp and the latter is even maintained independently: https://github.com/seccomp/libseccomp/releases
> Somehow we ended up with an overengineered mess of leaky abstractions for cloud-based, vendor-locked infrastructure.
Wait a moment - he cites Docker. That's owned by a private company. What does this have to do with Linux? If company xyz does something based on FreeBSD, we would then say company xyz is responsible for FreeBSD failing or not failing? How does that work?
> And this complexity has quietly reshaped how the industry thinks about deploying software. Today, if you want to run an application in a larger system, the implicit assumption is that you containerise it with Docker and orchestrate it with Kubernetes.
Personally I find all this abstraction crap. With all their failures, though, things such as docker kind of present a "download this one file, then it will work fine". And that is kind of true. I saw that in in-campus use for life science faculty clusters and what not. It simplifies things for the admin there. People give a similar rationale for systemd. Personally I don't think systemd should exist, but there are people who benefit from it - that simply is a factual statement.
All in all this is a very strange point of view from FreeBSD folks. At the least the NetBSD folks back then on the mailing list acknowledged the situation and then tried to find alternative strategies and in some ways succeeded (although I am not sure whether NetBSD right now runs on more toasters than Linux does - anyone has updated statistics for that?).
>>I think 500 out of 500 top supercomputers running Linux kind of show which philosophy is better.
Or is it because it's what they're used to. I saw this argument elsewhere where the respondent went on to show that the users were Linux specialists and that's why Linux was used.
FreeBSD doesn't have support for CUDA, many third party network fabrics and accelerators, Lustre, and other high performance storage solutions. It makes FreeBSD for frontier machines a non-starter. FreeBSD adoption for AI trainer/inference clusters has the same problems.
"failed to verify your browser"
Getting the same thing, "Failed to verify your browser. Code 11". Some noise about WebGL in the browser console, getExtension() invoked on a null reference. LibreWolf on Linux + resist fingerprinting.
Maybe opting for a better-written WAF could boost the reach?