The Frantic Week

If day zero was about getting a cluster to exist, the next ten days were about making it trustworthy enough to build on. This is the densest stretch of the entire project — over 130 commits in eleven days — and it’s where most of the patterns that hold for the rest of the migration were actually decided.

Secrets stop being sealed-secrets

On 7 April, sealed-secrets was replaced outright by External Secrets Operator backed by Vault — the same Vault inherited from the old world, reachable over the network the new cluster now shared with it. The switch wasn’t casual: the ClusterSecretStore was moved onto AppRole auth rather than a static token, ESO’s CRDs were applied with ServerSideApply because of their size, and — a detail that would matter for the rest of the project’s life — the API version was pinned. Getting the API version wrong for an ExternalSecret (v1beta1 vs v1) becomes a recurring, low-grade source of bugs for months afterward.

Two Kubernetes upgrades in two days

Before the platform had a single real application on it, it was upgraded twice: 1.33 → 1.34, then immediately 1.34 → 1.35. The second hop surfaced a genuine gotcha rather than a routine bump — Kubernetes 1.34 tightened PodSecurity admission enforcement, and the monitoring namespace’s baseline policy started blocking node-exporter, which legitimately needs hostNetwork, hostPID and hostPath to do its job. The fix was an explicit privileged PodSecurity label on the namespace — the first of many “this DaemonSet needs privileged, not baseline” exceptions carved out over the coming months.

An early node experiment, rolled back within the week

On 8 April, two new nodes were added to the cluster and briefly promoted to carry workloads. It didn’t last. By 16 April the commits read: “argocd: disable unused apps for talos-4/5 decom”, then “talos: remove talos-04/05 from repo, update docs and config for 3-node cluster”. In their place, the control-plane-only taint was lifted from the original three nodes instead — “talos: remove control-plane-only taint from talos-01/02/03 to allow workloads” — trading a hardware expansion that hadn’t proven itself yet for simply using the capacity already in the room. Node names get reused more than once later in this project’s history; this is the first instance of that pattern, not the last.

Storage and the first real outage-shaped headache

9 April brought Longhorn and the beginning of the PDNS stack in the same commit. PDNS turned out to be the harder half: importing an existing MariaDB schema hit IF NOT EXISTS syntax that MariaDB’s CREATE INDEX doesn’t support, then a version mismatch against the source server that needed pinning to MariaDB 10.11 specifically, then GTID replication settings, then a switch to internally-built PowerDNS Docker images with a daily CronJob to watch upstream for new releases. None of this was glamorous, but PDNS running correctly in-cluster from this point on is why the later Raspberry Pi migration (ns0/ns1/ns3) had a known-working config to start from.

The app landgrab

10–15 April is close to one app a day, sometimes more:

• Rocket.Chat was deployed and removed on the same day (10 April) — the only app in the whole project’s history to come and go within hours.
• Mattermost replaced it as the chosen chat platform, eventually becoming the alerting destination for both Alertmanager and Falco.
• Harbor landed as the container registry, immediately followed by a fight with its bundled Trivy scanner over CVE-2026-33634 — disabled, then reinstated as a standalone server pinned to a specific patched adapter version once the fix was confirmed.
• Gitea, with SSH and HTTPS sharing the cluster’s one gateway IP via Cilium’s LBIPAM annotation, plus Gitea Actions for CI.
• Loki + Alloy for log aggregation, and Trivy Operator for cluster-wide vulnerability scanning.
• Falco + Falcosidekick, alerting into Mattermost, with an early false positive on Cilium’s own CNI exec calls that needed a trusted_cni_exec macro to suppress.
• The registry mirror — an nginx pull-through cache in front of Docker Hub via Harbor — went in on 15 April specifically to stop the cluster hammering Docker Hub’s rate limits during all this churn.

docs/cluster-overview.md — the first attempt at a single inventory of everything running — was written on 13 April, already several apps behind what had shipped that same week.

Fixing the platform’s own blind spots

The week closes on 15 April with two fixes that aren’t about any application at all: kube-scheduler and kube-controller-manager were rebound to 0.0.0.0 and given static Endpoints objects so Prometheus could actually scrape them — Talos doesn’t expose their metrics ports externally by default, and without this fix the two most important control-plane components had been running invisible to monitoring the entire time. It’s a small thing, but it’s the first sign that “the platform” now included watching itself, not just running workloads.