← All chapters
Chapter 04 · 18 May – 25 June 2026

The GPU Detour

Not every chapter of this migration ends with something still running today. This one is the biggest, most technically involved detour in the whole project, and it ends with the cluster back exactly where it started: no GPU nodes at all.

A quiet false start

Ollama first appears on 18 May, running CPU-only on talos-04. It didn’t last two days — by 20 May it’s disabled again alongside UniFi, both moved into argocd/disabled/. Without a GPU behind it, CPU inference clearly wasn’t worth keeping powered on.

One very long day: 11 June

Three weeks later, talos-07 joins the cluster with a real GPU — an NVIDIA RTX 3080 — and everything happens on a single day, 11 June, in a sequence that’s worth reading in full because it captures exactly what GPU passthrough into a security-conscious, immutable OS actually costs in engineering time:

  1. NVIDIA extension names had to be corrected to their -production variants.
  2. The GPU taint needed setting via kubelet extraArgs, not registerWithTaints — the two aren’t interchangeable on Talos.
  3. Kernel modules for the NVIDIA driver had to be loaded explicitly via machine.kernel.modules — nothing about GPU support is automatic on an immutable OS with no package manager.
  4. The device plugin’s nodeAffinity needed fixing, then a RuntimeClass added, then a privileged namespace for its hostPath volumes.
  5. Auto-detection failed outright until the discovery strategy was set to nvml explicitly — then switched to cdi — then back to nvml plus privileged access to /dev/nvidiactl — then to cdi again. The commit log genuinely flips between the two strategies four separate times in one day.
  6. The NVML library needed mounting from the host, first to /usr/local/lib (wrong — collided with something already there), then to a dedicated /nvidia-libs path instead.
  7. The Helm chart itself was eventually abandoned in favour of a raw manifest, giving direct control over every one of the above.
  8. The final working configuration combines runtimeClassName: nvidia with an explicit NVIDIA_VISIBLE_DEVICES env var for full CDI injection, on device plugin v0.19.2.

The underlying failure mode behind several of these iterations was a glibc version mismatch between what Talos ships and what the NVIDIA userspace libraries expected — a hostPath mount of the host’s libraries straight into a container with an incompatible glibc doesn’t just fail to load, it segfaults. runtimeClassName: nvidia is what actually fixes that, by having the NVIDIA Container Toolkit inject a compatible library set at container start rather than relying on a raw bind-mount.

Once all of that landed, so did dcgm-exporter for Prometheus/Grafana GPU metrics, and Ollama started pulling qwen2.5-coder:14b on startup — the first point in this whole saga where something GPU-backed was actually working.

Models, mistakes, and a second card

The following two days show real usage, not just plumbing: llama3.2:3b added as a general-purpose model, a switch to gemma4:e4b — and then, on 13 June, gemma4:e4b-mlx added and reverted the same day once it became clear MLX is an Apple Silicon format, not something an NVIDIA card can run at all. gemma3:12b, a second qwen2.5-coder:14b, phi4 and llama3.1 round out the model list. On 17 June, models were pinned to stay resident in VRAM indefinitely and moved onto local NVMe storage on talos-07 via a dedicated hostPath PV, rather than the network storage everything else used — GPU inference cared about disk latency in a way nothing else in the cluster did.

A second card, an RTX 5090 on talos-08, joined on 19 June — new enough hardware (Blackwell architecture) that it needed a different driver package, nvidia-open-gpu-kernel-modules-production, rather than the one talos-07 used. Grafana grew a combined dashboard covering both cards, and Ollama itself migrated over to the newer, larger card.

The failure, and the honest retreat

The cracks show first in the docs, not the code: a troubleshooting section titled “GPU lockup + DCGM init 250” was added on 21 June, before the actual failure. The day after, talos-07 was removed from the cluster outright — “persistent GPU lockup” — real hardware instability, not a configuration problem. talos-04 rejoined to cover the lost capacity.

An ANPR (automatic number-plate recognition) app briefly rode along on talos-08’s remaining GPU on 23 June, complete with Telegram alerting — and had its GPU request removed the same day, moved to CPU-only, because the card was already fully committed to Ollama. One day later, on 24 June, both Ollama and ANPR were disabled and moved to argocd/disabled/, and talos-08 itself was removed from the cluster, replaced by talos-06. By 25 June, the ANPR manifests were deleted entirely and its namespace torn down.

What this chapter actually cost

Six weeks, two physical GPUs, a full day rebuilding the device plugin configuration from scratch multiple times, and in the end, nothing GPU-shaped survives in the cluster today. That’s not a failure to hide — it’s the honest outcome of trying real local LLM inference on hardware that turned out not to be reliable enough to keep, at a point where nothing else in the migration depended on it working. The device-plugin configuration that did eventually work is preserved in the docs precisely so that if GPU hardware ever comes back, this fight doesn’t have to happen twice.