The short version

The problem. Kubernetes keeps a service running while it is being updated, and it can fail in confusing ways. Knowing it means having watched it succeed and fail.

The answer. A small notes service with its own database, packaged with Helm and run in a three-node cluster on one machine. Around it are tests that install, update, roll back and remove it for real, and two failures caused on purpose.

The proof. While the service was being updated, a test program asked it for a response five times a second. Across four updates, 140 to 148 requests each, not one failed. When a deliberately broken release was pushed, the two healthy copies kept answering 749 requests in a row while the bad one sat stuck, and one command put things back.

What runs

The service (/notes, /healthz, /readyz, /version) takes and returns short notes. It runs as two identical copies behind one address, with a PostgreSQL database that keeps its data on a separate disk volume, in a local kind cluster, installed by one Helm chart. An update starts a new copy before it retires an old one, so there is never a gap. Around it sit the scripts and tests that bring the cluster up and down, build and load images (there is no registry), and exercise the release.

Versions used: kind v0.33.0, kubectl v1.36.1 (client), Helm v4.3.1, Radar v1.20.2, Python 3.12, uv 0.12.3, FastAPI 0.162.4, psycopg 3.3.8, PostgreSQL 17. The kind, helm and kubectl-radar binaries were downloaded as release binaries and checked against the checksum file published with the same release; there is no pipe-to-shell install.

Updates that drop nothing

A pod inside the cluster requested the service every 0.2 s while helm upgrade moved the Deployment from 0.1.0 to 0.2.0 (maxSurge: 1, maxUnavailable: 0). Four complete runs: 140, 148, 142 and 142 requests, 0 non-200 each, both versions seen. helm rollback returned the service to 0.1.0.

Getting that to hold took real debugging. The first update dropped one request out of 74, because an old copy stopped listening a moment before the cluster stopped sending it traffic. A 5-second preStop sleep on the app container fixed it; the next run was 133 requests, 0 non-200.

The probes say different things on purpose. With PostgreSQL stopped, /readyz answers 503 after about 3.5 s (the app’s connect timeout) while /healthz stays 200, so the service stops receiving traffic but is not restarted for no reason. The chart’s probe timeout is 5 s because of that gap.

The container runs as non-root with a read-only root filesystem, and the app works with both. PostgreSQL runs as a StatefulSet with a volume claim: a note written before the upgrade was still there after the rollback and after deleting postgres-0. The tests refuse to start over an existing release or data volume, because their cleanup deletes both.

Two failures, run on purpose

The log of both drills records the command, the symptom, the cause and the fix. The predictions made before each run were reasoned, not run, and are marked as they turned out.

A bad image mid-rollout. helm upgrade --set image.tag=9.9.9 --wait --timeout 60s exited 1 after the timeout. The new pod went ErrImagePull, then ImagePullBackOff; the two old pods kept serving. The request loop made 749 requests across the failed upgrade and the rollback, all 200, so maxUnavailable: 0 held. Radar’s Issues view showed the failed release and the pull error; its wording was “Not authorized to pull image”, while the event text says the repository does not exist. helm rollback reused the two old pods (same names, no restarts) and the Issues view cleared.

A reinstall over the old data volume. helm uninstall keeps the volume claim but deletes the Secret; the reinstall generated a new password while the old volume’s database still held the old one. PostgreSQL stayed Ready (its probe uses pg_isready, no login) and both app pods went CrashLoopBackOff with password authentication failed. The fix kept the data: set the new password from inside the database pod (its local socket trusts the connection), then restart the app. Radar showed three Critical issues and the pod logs matched kubectl logs. The chart was then changed so helm uninstall keeps the database Secret with its volume and a conflicting password is refused; the deploy test covers it.

A third check came true as predicted: changing the password value on a healthy install exits 0 and breaks nothing until the next pod start, and Radar flagged the stale Secret in an environment variable before that.

What went wrong while building

The failure log records every failure with cause and fix. A few examples:

• kind load rejected a multi-platform pulled image (content digest ... not found); the test now saves the image for one platform and loads the archive.
• Tests read /version while old pods were still terminating, so they saw the old version after the upgrade and after the rollback; they now wait until only two app pods remain.
• The deploy test could not start from a deleted cluster. The review session found this by reading the workflow, not by a run: an earlier fix had moved the preflight ahead of the script’s own cluster start. After the fix both cluster tests were run in the workflow’s order from no cluster.
• The integration tests were written after the code and passed on their first run; the log says so, because they were not watched failing first.
• The dashboard worded one cause in a misleading way, and the log notes that too.

Tests

Re-run on 2026-11-11: the app tests (20 passed) and the deploy test (22 checks) on the real cluster. The cluster test and the deploy test also passed on GitHub’s runner the same day.

The CI workflow

Three independent jobs on ubuntu-24.04 with timeouts and read-only permissions: app tests with uv, chart lint and template render, and a kind job that runs the cluster test and then the deploy test. Actions are pinned by commit SHA; kind and Helm are downloaded with their published checksums. The workflow parses and passes actionlint, and its commands were run locally in its order. It ran on GitHub on 2026-11-11 and all three jobs passed on the first run: app tests (20 passed, 23 s), chart lint and template (6 s), and the kind cluster job (5 min 18 s), whose deploy test recorded 134 requests with 0 non-200 during the rolling update.

Key numbers