Leaving leased address space, part 2: the rollback that gave away a promised IP

In October we moved the first batch of VMs off leased Cogent addresses and onto our own RIPE space, with one DHCP change per VM and no reboots. On 9 November 2026 at 00:00 UTC we moved the rest to another of our own prefixes.

It mostly went the same way. One thing did not, and it is the more useful part of this story.

What we changed after batch 1

More notice, and a usable address from day one. Customers got three emails: a month before, three and a half weeks before, and a few days before. The new address was reserved, routed and allowed in the VM's firewall from the first email. So anyone with a static config could switch early, test it, and be done before the cutover.

Our first notice said "do not configure the new IP yet". That was wrong: the address was already live. The second notice corrected it, and many customers switched early.

Old addresses stay for a few days. Same as batch 1: after the cutover the old address stays routed and allowed, so a customer can revert while fixing their side. We decided against forwarding the old address to the new one: it hides the problem instead of showing it.

The mistake

During the night one VM had trouble on its new address, and our cutover script rolled it back to the old one.

The rollback did what it was written to do: it moved the VM's lease back. It also released the VM's new address. Sixteen seconds later a new customer ordered a VM, and our provisioning gave that free address to them.

Hours later a follow-up email told the first customer to use a different new address, after three emails had promised the original one. The customer had already pointed their DNS at the promised address, and their services broke.

We fixed it the same morning by swapping the two VMs' addresses, so the first customer got exactly the address we had promised.

What we learned

• A promised address is a commitment. A rollback in a renumbering job must keep the new address held for that customer. Check that the hold is really there after the rollback, not only that the VM is back online.
• Every follow-up email must diff against what you already sent. No mail may name a different address than the earlier ones did, unless that change is the point of the mail.
• Provisioning does not pause for your maintenance. New orders keep coming during a cutover window. Anything you release, even for a second, can be taken.
• Routers throttle repeated logins. A bulk job that opens many SSH sessions can trip the router's login protection and slow down normal provisioning at the same time. Run bulk changes as one script in one session.

Where it stands

Every VM that lived on the leased Cogent range now has an address from a prefix we own and announce ourselves. The leased range will be returned once every last address on it is unused.