I feel vindicated :). We put in a lot of effort with great customers to get nested virtualization running well on GCE years ago, and I'm glad to hear AWS is coming around.
You can tell people to just do something else, there's probably a separate natural solution, etc. but sometimes you're willing to sacrifice some peak performance just have that uniformity of operations and control.
OCI supports it with Intel. I know it works with AMD, but we don't officially support that so far as I'm aware. The performance hit on AMD is bigger than Intel, last I looked.
Nowadays nested just wastes the extra operating system overhead and I/O performance if your VM doesn't have paravirtualization drivers installed. CPUs all have hardware support.
Is nested VMX virtualization in the Linux kernel really that stable?
The technical details are a lot more complex than most realize.
Single level VMX virtualization is relatively straightforward even if there are a lot of details to juggle with VMCS setup and handing exits.
Nested virtualization is a whole another animal as one now also has to handle not just the levels but many things the hardware normally does, plus juggling internal state during transitions between levels.
The LKML is filled with discussions and debates where very sharp contributors are trying to make sense of how it would work.
Amazon turning the feature on is one thing. It working 100% perfectly is quite another…
It's been around for almost 15 years and stable enough for several providers to roll it out in production the past 10 years (GCP and Azure in 2017).
AWS is just late to the game because they've rolled so much of their own stack instead of adapting open source solutions and contributing back to them.
> Nested virtualization is supported only on 8th generation Intel-based instance types (c8i, m8i, r8i, and their flex variants). When nested virtualization is enabled, Virtual Secure Mode (VSM) is automatically disabled for the instance.
This will make it easier to run automated tests in the Android emulator in CI using regular runners. It was a pain dealing with bare-metal instances just for that.
Support for nested virtualization has been added to the main SDKs. In the us-west-2 region, you can already see the "Nested Virtualization" option and use it with the new M8id, C8id, and R8id instance types.
This is really big news for micro-VM sandbox solutions like E2B, which I work on.
I wonder if this is connected to Azure launching OpenShift Virtualization on "Boost" SKUs? There are a lot of VMWare customers going to OpenShift Virt, and apparently the CPU/memory overhead on Azure maxes out around 10% under full load... but then hyper V has been doing a lot of work on it. No idea if nitro includes any of the KVM-on-KVM passthrough of full KVM, to give it an edge here.
Azure has had nested virt for a while - maybe it’s related to OpenShift but you could run OpenShift on Azure for some time. I used to run HyperV in Azure on certain SKUs
Yep. It's pretty boring. I've been using it at home for years and years with libvirt on very not-special consumer hardware. I guess the AWS clown is finally catching up on this one little not-new-at-all thing.
I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this.
If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it.
But EC2 is not like your home server. There are some nontrivial considerations and requirements to offer nested virtualization at cloud scale:
1. Ensuring virtualized networking (VPC) works with nested VMs as well as with the primary VM
Unfortunately I'm not at liberty to dive deep into those details. I will say that Firecracker can be used on bare metal EC2 instances, whether you're a public customer or AWS itself. :-)
Since I don't work for AWS I'm allowed to say that at the scale of millions/billions of microVMs you're better off running them on bare metal instances to avoid the overhead of nested virtualization.
Could someone explain why this is might be a big deal?
I remember playing with nested virty some years ago and deciding it is a backwards step except for PoC and the like. Given I haven't personally run out of virty gear, I never needed to do a PoC.
It is great for isolation. There are so many VM based containerization solutions at this point, like Kata Containers, gvisor, and Firecracker. With kata, your kubernetes pods run in isolated VMs. It also opens the door for live migration of apps between ec2 instances, making some kinds of maintenance easier when you have persistent workloads. Even if not for security, there are so many ways a workload can break a machine such that you need to reboot or replace (like detaching an ebs volume with a mounted xfs filesystem at the wrong moment).
The place I've probably wanted it the most though is in CI/CD systems: it's always been annoying to build and test system images in EC2 in a generic way.
It also allows for running other third party appliances unmodified in EC2.
But also, almost every other execution environment offers this: GCP, VMWare, KVM, etc, so it's frustrating that EC2 has only offered it on their bare metal instance types. When ec2 was using xen 10+ years ago, it made sense, but they've been on kvm since the inception of nitro.
One of the big benefits that gVisor offers is that it doesn't require nested virtualization (or any virtualization). They released a new version that improves performance when not using virtualization a while back: https://gvisor.dev/blog/2023/04/28/systrap-release/
You can now run VMs inside a cheaper AWS instance instead of having to pay for an entire bare-metal instance. This is useful for things like network simulation where you use QEMU to emulate network hardware.
If you have some workload that creates VMs, now you can run that workload on EC2 rather than having to use bare metal or some other provider that allows nested virtualization. There are many many such workloads. Just to give one example: testing a build system that spins up VMs to host CI jobs.
From memory, the virtualisation operations themselves aren't nested. The VM instructions interact with the external virtualisation hardware, so it's more of a cooperative situation, e.g. a guest can create & manage virtualisation structures that are run alongside it.
I don't know if this applies to the specific nested virtualisation AWS are providing though.
I feel vindicated :). We put in a lot of effort with great customers to get nested virtualization running well on GCE years ago, and I'm glad to hear AWS is coming around.
You can tell people to just do something else, there's probably a separate natural solution, etc. but sometimes you're willing to sacrifice some peak performance just have that uniformity of operations and control.
This is a big deal because you can now run Firecracker/other microVMs in an AWS VM instead of expensive AWS bare-metal instances.
GCP has had nested virtualization for a while.
OCI supports it with Intel. I know it works with AMD, but we don't officially support that so far as I'm aware. The performance hit on AMD is bigger than Intel, last I looked.
Azure has had nested virt available for a while too. I used to run HyperV in cloud
Was hoping this comment would be here. Firecracker and microVMs are good use-case. Also, being able to simply test and develop is a nice to have.
Nested virtualization can mean a lot of things. Not just full VMs.
> Firecracker and microVMs are good use-case.
Good use-case for what?
Nowadays universal answer for "what? why?" is AI. AI agent needs VMs to run generated code in sandbox as they can not be trusted.
I don't think everyone should assume that AI is the answer to all questions. I was asking the person I replied to, thanks.
whats the ~ perf hit of something like this?
Nowadays nested just wastes the extra operating system overhead and I/O performance if your VM doesn't have paravirtualization drivers installed. CPUs all have hardware support.
As a practical matter, anywhere from 5-15%.
Is nested VMX virtualization in the Linux kernel really that stable?
The technical details are a lot more complex than most realize.
Single level VMX virtualization is relatively straightforward even if there are a lot of details to juggle with VMCS setup and handing exits.
Nested virtualization is a whole another animal as one now also has to handle not just the levels but many things the hardware normally does, plus juggling internal state during transitions between levels.
The LKML is filled with discussions and debates where very sharp contributors are trying to make sense of how it would work.
Amazon turning the feature on is one thing. It working 100% perfectly is quite another…
It's been around for almost 15 years and stable enough for several providers to roll it out in production the past 10 years (GCP and Azure in 2017).
AWS is just late to the game because they've rolled so much of their own stack instead of adapting open source solutions and contributing back to them.
> Nested virtualization is supported only on 8th generation Intel-based instance types (c8i, m8i, r8i, and their flex variants). When nested virtualization is enabled, Virtual Secure Mode (VSM) is automatically disabled for the instance.
This will make it easier to run automated tests in the Android emulator in CI using regular runners. It was a pain dealing with bare-metal instances just for that.
Support for nested virtualization has been added to the main SDKs. In the us-west-2 region, you can already see the "Nested Virtualization" option and use it with the new M8id, C8id, and R8id instance types.
This is really big news for micro-VM sandbox solutions like E2B, which I work on.
I wonder if this is connected to Azure launching OpenShift Virtualization on "Boost" SKUs? There are a lot of VMWare customers going to OpenShift Virt, and apparently the CPU/memory overhead on Azure maxes out around 10% under full load... but then hyper V has been doing a lot of work on it. No idea if nitro includes any of the KVM-on-KVM passthrough of full KVM, to give it an edge here.
Azure has had nested virt for a while - maybe it’s related to OpenShift but you could run OpenShift on Azure for some time. I used to run HyperV in Azure on certain SKUs
Azure? OpenShift? "I don't think about you at all." — Matt Garman probably
welcome AWS to 2018!
Yep. It's pretty boring. I've been using it at home for years and years with libvirt on very not-special consumer hardware. I guess the AWS clown is finally catching up on this one little not-new-at-all thing.
I was an Amazon EC2 Specialist SA in a prior role, so I know a little about this.
If EC2 were like your home server, you might be right. And an EC2 bare metal instance is the closest approximation to that. On bare metal, you've always been free to run your own VMs, and we had some customers who rolled their own nested VM implementations on it.
But EC2 is not like your home server. There are some nontrivial considerations and requirements to offer nested virtualization at cloud scale:
1. Ensuring virtualized networking (VPC) works with nested VMs as well as with the primary VM
2. Making sure the environment (VMM etc) is sufficiently hardened to meet AWS's incredibly stringent security standards so that nesting doesn't pose unintended threats or weaken EC2's isolation properties. EC2 doesn't use libvirt or an off-the-shelf KVM. See https://youtu.be/cD1mNQ9YbeA?si=hcaZaV2W_hcEIn9L&t=1095 and https://youtu.be/hqqKi3E-oG8?si=liAfollyupYicc_L&t=501
3. Ensuring performance and reliability meets customer standards
4. Building a rock-solid control plane around it all
It's not a trivial matter of flipping a bit.
There's no better way to get good information that is right, than to say something that is misguided and/or wrong.
Thanks for the well-reasoned response.
I always enjoy the color you add to these conversations. Thanks!
I always enjoy the color you add to these conversations in your newsletter.
It's provided many a chuckle.
Thanks!
Seriously curious, don’t Firecracker VMs already run on EC2 instances under the hood when they host Lambda and Fargate?
Unfortunately I'm not at liberty to dive deep into those details. I will say that Firecracker can be used on bare metal EC2 instances, whether you're a public customer or AWS itself. :-)
I guess I should have peeked at the source code when I was there…
No need, at least when I was there when the day was still one, before the pandemic. And well, Firecracker is open source.
A few of the best technical presentations that I've watched were at a pre-SKO event. Nitro, Graviton and Firecracker.
Great engineering pieces, the three of them.
Since I don't work for AWS I'm allowed to say that at the scale of millions/billions of microVMs you're better off running them on bare metal instances to avoid the overhead of nested virtualization.
I used to work for AWS and I’m allowed to say the same thing. ;-)
Nitro is very interesting stuff
Would love to see performance numbers with nested virtualization, particularly that of IO-bound workloads.
Could someone explain why this is might be a big deal?
I remember playing with nested virty some years ago and deciding it is a backwards step except for PoC and the like. Given I haven't personally run out of virty gear, I never needed to do a PoC.
It is great for isolation. There are so many VM based containerization solutions at this point, like Kata Containers, gvisor, and Firecracker. With kata, your kubernetes pods run in isolated VMs. It also opens the door for live migration of apps between ec2 instances, making some kinds of maintenance easier when you have persistent workloads. Even if not for security, there are so many ways a workload can break a machine such that you need to reboot or replace (like detaching an ebs volume with a mounted xfs filesystem at the wrong moment).
The place I've probably wanted it the most though is in CI/CD systems: it's always been annoying to build and test system images in EC2 in a generic way.
It also allows for running other third party appliances unmodified in EC2.
But also, almost every other execution environment offers this: GCP, VMWare, KVM, etc, so it's frustrating that EC2 has only offered it on their bare metal instance types. When ec2 was using xen 10+ years ago, it made sense, but they've been on kvm since the inception of nitro.
One of the big benefits that gVisor offers is that it doesn't require nested virtualization (or any virtualization). They released a new version that improves performance when not using virtualization a while back: https://gvisor.dev/blog/2023/04/28/systrap-release/
When you run nested virt, you can do multicast in Cloud between the nested VMs. You can’t do multicast across VMs inside the Cloud.
Basically you setup a small LAN with HyperV or something similar (I have only done it with HyperV)
Hopefully it mean that you can finally run a network simulator like GNS3 https://www.gns3.com/ in an AWS instance.
You can now run VMs inside a cheaper AWS instance instead of having to pay for an entire bare-metal instance. This is useful for things like network simulation where you use QEMU to emulate network hardware.
If you have some workload that creates VMs, now you can run that workload on EC2 rather than having to use bare metal or some other provider that allows nested virtualization. There are many many such workloads. Just to give one example: testing a build system that spins up VMs to host CI jobs.
I wonder if providers like Hetzner and Digital Ocean etc. will get this someday also.
I wonder if this will extend SEV-SNP and TDX to the child VMs?
It says VSM is automatically disabled... so I would assume not.
Proof that we’re living in a simulation.
I haven't watched The Thirteenth Floor in a while. The kids today don't even know about it.
Would these thing be good for openclaw, agents?
Yeah, though honestly if I'm deploying anything I'd just build an image with nix rather than use nested virtualization.
Digital Ocean has always supported nested virtualization.
What's the performance impact for nested virtualization in general? I'd think this would be adding multiple layers of MMU overhead.
From memory, the virtualisation operations themselves aren't nested. The VM instructions interact with the external virtualisation hardware, so it's more of a cooperative situation, e.g. a guest can create & manage virtualisation structures that are run alongside it.
I don't know if this applies to the specific nested virtualisation AWS are providing though.
depends on the workload and how they've done it
pure CPU should be essentially unaffected, if they're not emulating the MMU/page tables in software
the difference in IO ranges from barely measurable to absolutely horrible, depending on their implementation
traps/vmexits have another layer to pass through (and back)
As a practical matter, anywhere from 5-15%.
"* *Feature*: Launching nested virtualization. This feature allows you to run nested VMs inside virtual (non-bare metal) EC2 instances."
Highly doubt that
Only took them 9 years. AWS so much innovation.
Remember, “customer obsession”.
But “protect revenue first”.
Sounds expensive for legacy apps
hell yes, finally