Hi,
Original propolis architect here, and primary engineer for much of the effort porting bhyve to illumos.
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
Not really familiar with bhyve or even illumos in the slightest. But reading this comment makes me a bit sad in a way. These types of conversations and comments used to be common where I worked just few years ago. Now no one even cares how something is implemented let alone a "I felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware". The "good" ones today are looking exclusively at external properties rather than internal ones. Thank you for the comment.
This brings back memories of watching someone do a live demo of bhyve at nycbug. I think that just have been almost a decade ago now, but the live demo was amazing.
Looks like this could really become a game changer for the FreeBSD folks using bhyve. The author often complained about this missing feature and he was also contributing to the Sylve project. Looks like he's crafting something.
I am using FreeBSD, not DragonflyBSD, but I happen to be on a DragonflyBSD developer mailing list, which is normally low traffic, except that I have noticed recently a lot of messages with various patches, mentioning both bug fixes and newly implemented features.
So the project is alive and it seems more active than previously.
There is some uptick in interest in various BSDs as a linux alternative given the rough few months it's been having. I know 2 companies that are trying to see if they can switch some edge nginx proxies to run on OpenBSD. A RCE in a TCP handshake is up there tbf
I love FreeBSD. But I reluctantly had to migrate one of my home servers to Ubuntu Server because I needed Docker support as well as ZFS. Ubuntu Server, compared to FreeBSD, is a complete clusterfuck of an OS. And I regret the effort I spent migrating.
There's experimental support for Podman and OCI containers on FreeBSD 14.3 and later, with some additional system configuration required [0], [1]. I've been playing with it off and on with FreeBSD 15.0 and 15.1, but I'm not using it for anything I depend on.
I used to use Ubuntu a lot about 7 years ago. I saw the writing on the wall. Once they started forcing snap down people’s throats, especially. I’ve been on Debian ever since and have been very happy. I also have run many other OS’s and I feel that Debian is still a good blend of polished and still friendly to the hacker.
Yeah Debian is nice. But at the time of switching, only Ubuntu offered ZFS precompiled in their official repos.
I did, for a period, run some other distro with DKMS but that was a massive pain to keep up-to-date due to ZoL’s piss poor documentation and messy build tools.
This was quite a few years ago though. Long before even FreeBSD synced with ZoL for OpenZFS. So the problems I encountered have likely matured tenfold since.
But if I’m going to the hassle of installing a new OS then I might as well install FreeBSD again because I just prefer it to Linux (nothing against Linux in general, my laptops and PCs all run Arch and I’ve made a career from supporting a variety Linux platforms)
I did consider the bhyve approach myself, before I made the switch. But it was still pretty experimental at the time (iirc) and wanted something reliable otherwise I’d have a wife and two kids moaning at me that their stuff isn’t working haha!
But maybe you’re right that it’s worth reconsidering now.
Also related is Propolis from Oxide who worked on live migration with Bhyve https://github.com/oxidecomputer/propolis
Hi, Original propolis architect here, and primary engineer for much of the effort porting bhyve to illumos.
Quite a bit was changed between "illumos bhyve" and FreeBSD upstream. We in illumos decided to take a slightly different approach in how we approached the machinery for saving/restoring the state of devices emulated in-kernel, which is a necessary part of doing such a live migration. FreeBSD was first to add that functionality, but we (I) felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware. It was at that point that illumos "soft-forked" the kernel VMM component so we could take a different approach. While the two kernel VMM components are very similar, and much of the bhyve userspace emulation (a counterpart to propolis) will run on either OS, there are some key differences around instruction emulation and those kernel-emulated devices which prevent them from being drop-in replacements for each other.
All that said, I'm happy to see FreeBSD making progress on this front. They've been a friendly peer in this space, and both projects benefit from efforts in this area.
Not really familiar with bhyve or even illumos in the slightest. But reading this comment makes me a bit sad in a way. These types of conversations and comments used to be common where I worked just few years ago. Now no one even cares how something is implemented let alone a "I felt it exposed too much in the way of implementation details from that device emulation, rather than a more abstract model of the emulated hardware". The "good" ones today are looking exclusively at external properties rather than internal ones. Thank you for the comment.
This brings back memories of watching someone do a live demo of bhyve at nycbug. I think that just have been almost a decade ago now, but the live demo was amazing.
Looks like this could really become a game changer for the FreeBSD folks using bhyve. The author often complained about this missing feature and he was also contributing to the Sylve project. Looks like he's crafting something.
Anyone heard anything from dragonflyBSD lately? This sounds like something that might go well with HammerFS?
I am using FreeBSD, not DragonflyBSD, but I happen to be on a DragonflyBSD developer mailing list, which is normally low traffic, except that I have noticed recently a lot of messages with various patches, mentioning both bug fixes and newly implemented features.
So the project is alive and it seems more active than previously.
There is some uptick in interest in various BSDs as a linux alternative given the rough few months it's been having. I know 2 companies that are trying to see if they can switch some edge nginx proxies to run on OpenBSD. A RCE in a TCP handshake is up there tbf
What’s FreeBSDs story like for Docker these days?
I love FreeBSD. But I reluctantly had to migrate one of my home servers to Ubuntu Server because I needed Docker support as well as ZFS. Ubuntu Server, compared to FreeBSD, is a complete clusterfuck of an OS. And I regret the effort I spent migrating.
There's experimental support for Podman and OCI containers on FreeBSD 14.3 and later, with some additional system configuration required [0], [1]. I've been playing with it off and on with FreeBSD 15.0 and 15.1, but I'm not using it for anything I depend on.
[0]: https://docs.freebsd.org/en/books/handbook/containers/
[1]: https://podman.io/docs/installation#installing-on-freebsd
Nice. I’ll have a read of that later. Thank you
There is also this which was not mentioned by other commenters.
https://daemonless.io/
Looks interesting. Thanks for sharing.
I used to use Ubuntu a lot about 7 years ago. I saw the writing on the wall. Once they started forcing snap down people’s throats, especially. I’ve been on Debian ever since and have been very happy. I also have run many other OS’s and I feel that Debian is still a good blend of polished and still friendly to the hacker.
Yeah Debian is nice. But at the time of switching, only Ubuntu offered ZFS precompiled in their official repos.
I did, for a period, run some other distro with DKMS but that was a massive pain to keep up-to-date due to ZoL’s piss poor documentation and messy build tools.
This was quite a few years ago though. Long before even FreeBSD synced with ZoL for OpenZFS. So the problems I encountered have likely matured tenfold since.
But if I’m going to the hassle of installing a new OS then I might as well install FreeBSD again because I just prefer it to Linux (nothing against Linux in general, my laptops and PCs all run Arch and I’ve made a career from supporting a variety Linux platforms)
> What’s FreeBSDs story like for Docker these days?
My current setup for Docker (Podman really in my case):
My plan: So basically just ditching Proxmox for FreeBSD/bhyve. Then at long last I'll be, again, systemd-free.I did consider the bhyve approach myself, before I made the switch. But it was still pretty experimental at the time (iirc) and wanted something reliable otherwise I’d have a wife and two kids moaning at me that their stuff isn’t working haha!
But maybe you’re right that it’s worth reconsidering now.
Nice! Can we have high availability / clustering for jails next?
That sounds interesting. Could you elaborate what that would mean?
so bhyve getting live migration finally means sylve might actually be usable for production clusters, been waiting on this.