Cutting the Cord

This is the chapter where “migration” stops being an ongoing project and starts being a series of final, irreversible steps. Three weeks, and by the end of them almost nothing from the old world is still load-bearing.

Finishing what July started

The week opens tidying up the Longhorn rollout from the previous chapter: GitLab’s backup job was hitting OOMKills and got its memory limit raised to 4Gi (27 July), and the remaining stateful PVCs — GitLab’s Gitaly repo storage, Harbor’s job service and registry, and the SeaweedFS object store GitLab’s backups deliberately avoided — all made the same pause-migrate-resume trip onto Longhorn on 28 July.

Two real bugs surface in the days after, both worth naming because they’re the kind of thing that only shows up under real load. On 29 July, the imapsync CronWorkflow started intermittently failing its OpenBao AppRole logins — the fix was serializing the workflow so 39 accounts didn’t all try to authenticate to OpenBao in the same instant and trip its rate limiting. The same day, talos-04 and talos-06 fell into a kubelet restart loop: their Cilium Egress Gateway IPs shared the same /24 as the node’s own IP, which confused the kubelet’s NodeIPController into thinking its own address kept changing. The fix was pinning each node’s nodeIP explicitly to exclude the egress addresses.

2 August: the last VMs

2 August 2026 is the day the cluster’s own control plane stopped being partly virtual. talos-01 and talos-03 — the original VM-based, control-plane-only nodes going all the way back to the cluster’s first talconfig.yaml — were removed for good. Every control-plane node from this point on is physical, schedulable hardware.

The same day carries UniFi’s real migration — not the false start from chapter 3, this time done properly: Longhorn storage instead of NFS, a Mongo dbOwner grant fixed on the restore/backup staging databases, Mongo’s memory limit raised twice (first to some initial bump, then to 8Gi) after it got OOMKilled mid-restore, a Recreate deployment strategy adopted specifically because UniFi’s RWO PVCs deadlock under the default rolling-update strategy if the replacement pod lands on a different node, and — replacing the old VM’s own automation — a proper rclone-based sync of UniFi’s daily backup straight to the real Dropbox.com account, after an NFS-based attempt turned out not to actually replicate what the VM had been doing. A full writeup landed in the docs the same day.

The inherited Vault, switched off

4 August carries one line that closes a chapter going back to before this repository even existed: “docs: mark old Vault cluster as fully decommissioned.” The Vault cluster spread across srv2.8p.uk and cp03.abt.me, with debian-gitrunner as its original Raft leader — the exact secrets infrastructure described in the very first entry of this journey — is finally, fully torn down. Package, data, TLS material, config, and the dedicated WireGuard mesh those nodes used to reach debian-gitrunner all go with it. debian-gitrunner itself stays powered off, untouched, a museum piece rather than an active dependency.

More node churn, and a second attempt at physical growth

Node topology keeps moving through the rest of this stretch: talos-03 rejoins on 8 August as a genuinely new fifth physical node (the name reused a third time in this project’s history), surfaces a recurring Longhorn disk-creation race on join that gets documented rather than fixed outright, then is removed again on 11 August along with talos-02, settling the cluster back to the three-node shape it holds today. A second in-cluster GitLab Runner is added the same day, this one specifically registered against gitlab.com rather than the self-hosted instance.

The GitLab.com pivot

By 13 August, CI jobs start repointing at registry.abt.me, the project’s new self-hosted image registry, in place of the in-cluster one. Then, on 14 August, the whole cluster’s own CI/CD and ArgoCD source moves to gitlab.com (abtme1/talos) — the repository this very account is written from switches home mid-project.

The following day brings the migration’s most serious real incident: an accidental OpenBao root token revoke during cleanup work. Recovery took most of a day and is documented in full as its own RCA rather than glossed over. GitLab itself was paused properly that same day — scaled down through its own Helm values rather than a live kubectl scale, so the desired state in git stayed truthful — ahead of its final removal.

16 August: “Decommission in-cluster GitLab.” The instance built on 19 July, cut over on 20 July, is gone less than a month after it went live, fully superseded by gitlab.com. The cleanup wasn’t perfectly clean — a ReferenceGrant got deleted along with the rest of GitLab’s manifests that shouldn’t have been coupled to them, and notification.turnbull.uk returned 500s for several hours before the missing cross-namespace grant was spotted and restored. It’s included here rather than left out, because an honest account of a migration includes the mistakes that happened in production, not just the ones caught before they shipped.

By the end of this chapter, the picture has fully inverted from where chapter zero started: no VM carries control-plane responsibility, the inherited Vault is gone, UniFi runs properly in-cluster, and even the project’s own source of truth has moved to infrastructure outside the home lab entirely.