Skip to content

Renumbering customer VMs in one night, without a reboot

Part of our Lisbon address space was leased from Cogent. Leased addresses work, but they are not yours: the carrier owns them, and leaving means renumbering every server on them. So we are moving customers onto our own RIPE prefixes, announced from our own network, AS199053.

On 1 October 2026 at 00:00 UTC we moved the first batch of customer VMs to a new /24.

The goal: one change per VM

Renumbering usually means logging into every server, editing its network config and rebooting. Every customer is one more chance to break something we cannot even see.

We wanted the night itself to be boring. So all the work was done weeks before, and the cutover was one small change per VM.

Weeks before: prepare everything

1. Get the prefix live first. The new /24 was registered, signed (RPKI ROA), given reverse DNS and announced weeks before anyone used it. New addresses need time to propagate and to be accepted by upstream filters.

2. Reserve every target address. Each VM got its new address assigned in advance. On the router, that address was held by a DHCP reservation with a placeholder MAC, so nothing else could take it in the meantime. A paper mapping in a spreadsheet does not protect an address; a real lease does.

3. Allow both addresses in the firewall. Every VM's anti-spoofing filter on the hypervisor got the new address next to the old one. From that moment the VM was allowed to use either.

4. Tell customers early. Notices went out more than four weeks ahead, with a reminder two weeks later. Each mail listed the customer's VMs, the old address and the new one.

The night: a DHCP lease rewrite

With all of that in place, moving a VM was one change: its DHCP lease now points to the new address.

Our DHCP leases are short (10 minutes), so each guest picked up its new address on its own at the next renewal. No login into customer servers, no config edits, no reboot.

Keep the old address alive

We did not remove the old addresses that night. They stayed in each VM's firewall filter and stayed routed for days. A customer whose service broke could set the old address back and fix things at their own pace, instead of being offline while they read our email.

What we learned

  • Customers think in subnets, not in your batches. Our split into batches was internal bookkeeping. A customer with VMs in both batches got one notice, counted his servers and asked where the last one was. He was right. Notify per customer, listing every VM they own in the old range.
  • Static configs do not follow DHCP. Customers who had set their address statically inside the VM had to change it themselves. The notices said so clearly, and the old address kept working for those who were late.
  • Do it all on the router in one script. A few hundred separate SSH commands to a router is slow and fragile. One uploaded script is one session and one result.
  • Moves break whitelists you forgot about. Our own servers that moved earlier broke firewall rules elsewhere that named their old address. Search every whitelist for the old address before you move.

The second batch followed five weeks later. That story, including the mistake we made, is in the second part.