GitOps Discipline: Storage, Secrets, and a Self-Hosted GitLab

After the GPU detour wound down, the project returns to infrastructure that actually mattered for the long haul. This ten-day stretch is, by commit density, the single busiest week of the entire migration — and it’s where the platform’s two hardest problems, storage and secrets, both finally got solved properly.

Warming up: egress and workflows

Two smaller pieces set the stage. On 13–14 July, per-WAN blackbox-exporter probes were wired up through dedicated Cilium Egress Gateway IPs — the origin of the .76–.79 reserved block that stays carved out of the general LB pool to this day. On 15 July, Argo Workflows was introduced for exactly one reason: replacing 39 separate imapsync CronJobs with a single CronWorkflow. Thirty-nine individual scheduled jobs polling IMAP accounts was never going to scale as a maintenance burden — one workflow definition parameterised over 39 accounts was.

A GitLab instance that lasts less than a month

On 19 July, a fresh self-hosted GitLab EE instance goes in at git.turnbull.uk, and the same day turns into a small networking saga of its own: a dedicated Gateway and IP were tried first, then reverted when it became clear Cilium doesn’t support TCPRoute for the SSH side of Git traffic — the fix that actually stuck was a small dedicated L4 proxy so GitLab could serve both web and SSH off one shared IP.

The next day, 20 July, GitLab was cut over onto the reused .110 IP — gitlab.turnbull.uk pointing at this new in-cluster instance instead of wherever it had lived before — with nightly backups (database, repos, uploads, artifacts, deliberately kept off the SeaweedFS object store it also depended on) and a Telegram notification on success or failure. Zabbix was rebuilt as a proper ArgoCD app the same day and cut over onto its own reused IP, .150.

It’s worth naming plainly what this instance’s fate turns out to be, because it colours everything else in this chapter: this GitLab, built on 19 July, is decommissioned less than a month later, on 16 August, once gitlab.com takes over as the project’s real CI/CD home. Nothing here was wasted — the backup discipline and networking lessons carried forward — but it’s a reminder that not every piece of infrastructure built during this migration was meant to be permanent.

Longhorn, done properly this time

Longhorn had already been tried and torn out twice earlier in the project (see chapters 2 and 3). On 21 July, it goes in a third time, and this time it sticks — scoped deliberately to talos-04/05/06 rather than the whole cluster, with node labels, disk-config annotations and kubelet extra mounts prepared on each node first.

What follows is a genuine migration marathon, one stateful app at a time, all in a single day: Authentik’s PostgreSQL volume, Kanboard’s data volume, GitLab’s own PostgreSQL and Redis, Harbor’s database, Redis and Trivy stores, Zabbix’s PostgreSQL, and the entire Prometheus/Alertmanager/Grafana stack. Each one follows the same disciplined sequence — pause the app’s ArgoCD auto-sync, migrate the underlying PVC onto Longhorn, resume auto-sync — so that no app is left half-migrated if something goes wrong partway through. A few ArgoCD drift issues surface along the way (Longhorn’s own CRD conversion config, a leftover controller-gen annotation) and get folded straight into the standard ignoreDifferences pattern established back on day one.

The actual secrets cutover

The week closes on 22 July with the change that mattered most for everything that came after: docs: add openbao-setup.md, mark vault-ha-setup.md as fallback cluster. This is the real Vault-to-OpenBao cutover — the inherited Vault cluster from the old world, running on srv2.8p.uk and cp03.abt.me since before this repository existed, handed off to a new three-node OpenBao Raft cluster on dedicated Raspberry Pi hardware. The backup tooling was renamed in the same spirit as everything else that week — vault-backup became openbao-backup, a logical rename only, deliberately not touching the namespace or PVC names underneath it, so nothing downstream that referenced the old names silently broke. The old Vault cluster wasn’t torn down yet — that’s a separate, later step — but as of this day, it’s a fallback, not the system of record.

By the end of this week, the two hardest infrastructure problems in the whole migration — durable storage and durable secrets — were both solved for good. Everything that follows is either extending that foundation to more of the old world, or cleaning up what’s left of it.