Skip to content

Patching every hypervisor in one day: rolling kernel updates for CVE-2026-99999

CVE-2026-99999 is a flaw in the Linux kernel's IPv4 stack, rated CVSS 9.3. It can let a virtual machine break out to its host. On a VPS platform every customer is root inside a guest, so every host was exposed.

There was no workaround. The only fix was the patched kernel and a reboot of every hypervisor.

What we told customers

We are patching every hypervisor against CVE-2026-99999, a critical Linux kernel flaw that can let a VM break out to its host. We move VMs to already-patched hosts before each host reboots, so most of you will notice nothing. Some VMs may see a short pause or a brief connection drop during the move.

The plan

  1. Sweep. Check the running kernel on every node. Result: every host was vulnerable.
  2. Patch the spares first. We powered on our standby nodes, upgraded them and rebooted them onto the fixed kernel while they were still empty. No VM was ever moved onto an unpatched host.
  3. Drain, reboot, return. Live-migrate every VM off a node onto patched hosts, reboot the empty node onto the new kernel, check uname -r, then bring its VMs back.
  4. Repeat node by node, with the drained and patched nodes becoming targets for the next ones.

All reachable nodes ran the fixed kernel by the early hours of the next day.

What broke, and what we learned

These are the things that cost us time. If you run Proxmox yourself, they will probably cost you time too.

Local disks make live migration slow

Most of our VMs live on local ZFS. Live-migrating them means copying the whole disk over the network (--with-local-disks), which takes minutes per VM. VMs on Ceph move in seconds, because only the RAM is copied. During the incident we moved some large, quiet disks to Ceph in advance so their migration became RAM-only.

Snapshots on local disk block live migration

A VM with snapshots on local storage cannot be live-migrated. It needs replication first, or an offline move.

Cloud-init snippets do not travel

Per-VM cicustom snippet files and hook scripts on local storage are not moved by migration. Copy /var/lib/vz/snippets/ to every target node, or the VM loses its cloud-init config on its next cold start. A cloud-init drive on local also needs an explicit storage map: --targetstorage zfsa:zfsa,local:local.

Corosync config_version races

When one node reboots while another node joins the cluster, the rebooting node can come back with an older config_version and corosync exits:

Received config version (67) is different than my config version (66)! Exiting

The node shows grey in the GUI. Copy the current corosync.conf from a healthy node and restart corosync and pve-cluster. After every reboot, check systemctl is-active corosync.

The enterprise repository on unsubscribed nodes

Spare nodes had pve-enterprise.sources enabled with no subscription, so apt full-upgrade failed with HTTP 401. Disable that source before the upgrade, and only reboot automatically if the upgrade succeeded.

Never stop a running migration by killing its unit

Stopping the job that runs a migration leaves a mess behind: a lock on the VM, a half-copied disk on the target and a block job inside QEMU. Our drain scripts stop only on a flag file, after the current VM finishes.

Some VMs had to ride the reboot

A small number of VMs failed live migration with a QEMU block-export-add error. They stayed on their host and were restarted with it, about five minutes of downtime each.

What we changed afterwards

The biggest lesson is storage. On shared storage a whole host can be emptied in minutes, not hours. New VMs are moving towards Ceph, and existing quiet VMs follow as capacity grows.