A enjoyable venture: microvms with Incus

I noticed this blog post for microvms in Proxmox, and it acquired me pondering, “Couldn’t I do the same in Incus?” So I made a decision to fiddle and see if I might find yourself with a profile I might make, to utilize qemu’s microvm characteristic in Incus. It seems, it’s solely doable to get it working, and it really works higher than I might have imagined.

What’s a microvm?

Here is QEMU’s own description:

microvm is a machine type inspired by Firecracker and constructed after its machine model. It’s a minimalist machine type without PCI nor ACPI support, designed for short-lived guests. microvm also establishes a baseline for benchmarking and optimizing both QEMU and guest operating systems, since it is optimized for both boot time and footprint.

A regular Incus VM is a full emulated machine: a q35 chipset, a PCIe topology, ACPI, and UEFI firmware (OVMF) that runs a bootloader off the guest disk, which then loads the guest’s own kernel and initramfs. It behaves like a physical PC. A microvm throws nearly all of that away. There is no firmware in the boot path at all, so QEMU is handed a kernel directly and jumps straight into it. The result sits much closer to a container in overhead, while still being a real, isolated VM with its own kernel.

One thing surprised me, and it is worth clearing up because QEMU’s wording is a little misleading. The definition says “without PCI nor ACPI,” but that describes the bare default (pure virtio-mmio, the Firecracker-like model). The microvm machine type also has opt-in pcie=on and acpi=on switches. I turn both on. That is deliberate: PCI is what lets Incus hot-plug a normal NIC into the guest, and it is also what made the SR-IOV experiment below work. So you get to choose where on the minimalism spectrum you want to sit.

The catch: you supply the kernel

Because there is no firmware and no bootloader, the guest’s own kernel never runs. You hand QEMU a kernel with -kernel, and that kernel has to be able to mount any guest’s root filesystem and bring up the Incus agent entirely on its own.

So I built a small, monolithic kernel (CONFIG_MODULES=n, everything compiled in, no /lib/modules needed in the guest at all) tuned for my CPU. The rule is that everything the guest could need has to be built in (=y), because there is no initramfs to load modules from:

• virtio blk / net / scsi / console / balloon / rng, plus VIRTIO_PCI
• 9p + virtio-9p + vsock + virtio-vsock (this is what the Incus agent rides on; miss it and the VM boots fine but the agent never appears)
• ext4, vfat, overlayfs, virtiofs
• the netfilter stack, TCP BBR, and so on
• BPF_SYSCALL (modern systemd wants it)
• and, just to see if it would work, the iavf driver for Intel SR-IOV virtual functions

Stripped down from a distro defconfig it lands around 12 MiB and boots a guest to userspace in roughly a quarter of a second.

The profile

This is where Incus really shines. raw.qemu.scriptlet lets you rewrite the QEMU configuration in Starlark before the VM launches. I use it to switch the machine type to microvm and drop the firmware bits Incus adds for q35:

def qemu_hook(instance, stage):
    if stage == "config":
        out = []
        for s in get_qemu_conf():
            n, e = s["name"], s["entries"]
            if n == "machine":
                e["type"] = "microvm"
                e["acpi"] = "on"; e["pcie"] = "on"; e["rtc"] = "on"
            elif n == "drive" and e.get("if") == "pflash":
                continue          # no OVMF
            elif n == "global" and e.get("driver") == "ICH9-LPC":
                continue          # q35 southbridge, gone
            out.append(s)
        set_qemu_conf(out)

The scriptlet can rewrite the machine config, but it cannot add -kernel. That goes in plain raw.qemu. Here is the whole profile:

config:
  security.secureboot: "false"
  limits.cpu: sockets=1,cores=4,threads=1
  limits.memory: 2GiB
  raw.qemu: -kernel /path/to/microvm-kernel -append "root=/dev/vda2 rootwait rw console=ttyS0"
  raw.qemu.scriptlet: |-
    ... the Starlark above ...
devices:
  eth0:
    type: nic
    nictype: bridged
    parent: br0
  root:
    type: disk
    path: /
    pool: default
    io.bus: virtio-blk   # makes the disk show up as /dev/vda

And then it is just:

incus launch images:debian/13 test --vm -p microvm

The gotchas

None of these were obvious, so I am writing them down for anyone who wants to try this:

1. CPU hotplug. Incus always asks QEMU about hot-pluggable CPUs, and microvm does not support that, so the VM dies on start. The fix is to set limits.cpu as a topology (sockets=1,cores=4,threads=1) rather than a plain number. A fixed topology disables hotplug and the error goes away.

2. Secure Boot. Incus refuses to create the VM (“incompatible with secureboot”) before the scriptlet ever runs. Since there is no OVMF here anyway, security.secureboot=false is the correct setting, not a compromise.

3. ro vs rw, the sneaky one. With root=... ro in the append, Alpine and Debian were fine, but Arch came up with no networking and no DNS. The cause: Arch’s cloud image ships with no /etc/machine-id and relies on systemd generating one on first boot. At that moment /etc is still read-only, so systemd cannot write it, and with no machine-id it cannot build a DHCP DUID, so systemd-networkd fails the whole link. A normal VM dodges this because its initramfs has root writable in time. Booting rw fixes it cleanly, and Arch then generates its own machine-id on first boot like it should.

4. BPF_SYSCALL. Without it, systemd logs Unable to load sysctl monitor BPF program: Function not implemented. Not fatal, but build it in to keep modern systemd happy.

Does one profile really boot anything?

That was the real question: one profile, or one profile per image? It turns out to be genuinely one profile. With the same microvm profile I launched Alpine, Arch, and Debian/trixie, all unmodified, and all three boot, get an IP, and resolve DNS with no per-image tweaks. Even better, the incus agent works just like a normal VM. The only assumptions are an ext4 root on partition 2 (every Incus image I tried uses that) and the matching filesystem built into the kernel. Because you bypass the image’s own initramfs, anything that depends on it (LUKS, LVM-on-root) will not work, but ordinary cloud images just go.

The numbers

Same Alpine image, same 1 GiB of RAM, idle just after boot:

Boot time, measured as incus start to the agent answering, warm restarts, same Alpine image:

The memory saving is the headline here: roughly half the host RAM for the same guest. With the high cost of RAM lately, this matters, and it matters more the more guests you pack onto a box.

The boot speedup is real too, but more modest, around three seconds. It is worth being precise about where it comes from, because it is not just skipping the bootloader. A normal VM spends time in the OVMF firmware, then the bootloader, then the kernel’s own initramfs, all before userspace even starts. The microvm skips that entire chain and runs a smaller kernel that initializes in about a quarter of a second. Once the distro’s userspace and the Incus agent come up, the two paths are identical, which is why the gap is a few seconds rather than tenfold. For a single VM that is minor; for a fleet of short-lived guests it adds up, and the RAM difference adds up a good deal faster.

Bonus: SR-IOV works too

Since I had built the iavf driver into the kernel, I tried attaching an SR-IOV virtual function from a physical NIC to one of these microvms, with a VLAN set on the VF. It just worked. The host hands the VF through, the guest drives the passed-through NIC natively with iavf, and the link comes up tagged on the correct VLAN. So these are not just toys; they can have real, line-rate hardware networking.

This is also why I keep PCI enabled. I did consider the pure virtio-mmio (no-PCI) variant, which is the most minimal microvm of all. But when I actually measured it, the difference was tiny: roughly 2 MiB less host RAM and only a couple dozen milliseconds faster to boot. That is not worth giving up PCI for, because PCI is what makes device passthrough and NIC hot-plug possible in the first place. So I leave pcie=on and get the passthrough abilities essentially for free.

What I have not tried (and probably will not work)

To be clear about scope: I only ran this on a single host in a homelab. I have not touched any of the cluster-oriented features, and I would expect most of them not to work as-is. Live migration, cluster scheduling and evacuation, and anything that assumes a standard machine type or a stable firmware and CPU model across hosts all seem unlikely to behave. Snapshots and config copies of the instance should be fine, but I have not stress-tested those either. Treat this as “fun and genuinely useful on one box,” not “production cluster-ready.”

I would love to see this officially supported

What strikes me most is how little it took: a small kernel and a short scriptlet, and you end up with what feels like a real middle ground between a container and a VM. A true, isolated guest with its own kernel, but at close-to-container overhead and boot time. I would love to see microvm become a first-class, supported option in Incus, because from the outside it does not look like it would be hard to add.

And here is the best part: none of this requires patching or modifying Incus at all. Everything above is just configuration and a scriptlet on a stock Incus. So even though it is not official, it is already entirely available to anyone who wants to do it today.

What I plan to do next

I’m currently working on a way to convert OCI images and container rootfs to microvms that work in Incus. If I’m able to accomplish all that, I’m considering proposing it as a new feature for future versions of Incus, if that’s a direction the project is interested in going.

Happy to share more info if anyone has any questions or needs help.