Cruxly › Cloud and DevOps › Cloud & DevOps: Containers and Orchestration

Liveness, Readiness, and a Rollout That Doesn't Take the Service Down

the probe misconfiguration that causes its own outage

Prompt: design the health-check and rollout strategy for a service with a slow startup (it needs ~20 seconds to warm a cache before it can serve traffic correctly) that gets deployed several times a day. The rollout must never send traffic to a pod that isn't actually ready, and a slow dependency at startup must never cause a healthy pod to be killed and restarted in a loop.

30 minutes first. The naive version uses one probe for everything and picks a timeout that "seems fine" — this is the single most common self-inflicted outage in a Kubernetes deployment.

Liveness and readiness ask different questions

Liveness answers "is this process stuck and should be killed and restarted" — it exists to recover from deadlocks and unrecoverable internal states, not to signal "not ready yet." A liveness probe that fails during normal slow startup causes Kubernetes to kill and restart a pod that was never actually broken, just still warming up. The time before liveness kills a container is roughly initialDelaySeconds + periodSeconds × failureThreshold — with periodSeconds: 3, failureThreshold: 3 and no delay, that's about 9 seconds. If startup always takes 20 seconds, the pod is killed around second 9, restarts, takes 20 seconds again, is killed again: a crash loop caused entirely by the probe. (timeoutSeconds is a different field — how long one probe request may take to answer.)

Readiness answers "should this pod currently receive traffic" — it can flip to "not ready" temporarily (warming a cache, a downstream dependency being briefly unavailable) without the pod being killed. A pod that fails readiness is simply removed from the Service's load-balancing rotation until it passes again; nothing is restarted. This is the mechanism that should handle "still warming up," not liveness.

Using the same probe config for both is the root cause of most probe-induced outages. A timeout tuned to catch real deadlocks quickly (short) is too aggressive for readiness during legitimate slow startup; a timeout generous enough for slow startup (long) is too slow to catch an actually-stuck process via liveness. They need different timing — periodSeconds and failureThreshold, which together set how long a failure lasts before action — because they answer different questions on different timescales.

startupProbe exists specifically for this

A startupProbe disables liveness and readiness checks until it itself succeeds once, which is the direct fix for "slow startup shouldn't look like a liveness failure." Give the startup probe a generous timeout and failure threshold appropriate to the slowest realistic startup (with margin), and only once it passes do the normal liveness/readiness probes with their own, tighter timeouts take over.

Without a startup probe, the only way to avoid a startup-induced crash loop is to set the liveness probe's own timeout as generously as startup needs — which then makes liveness slow to catch a real deadlock later, during normal operation, long after startup finished. The startup probe exists precisely to decouple these two unrelated timing requirements instead of forcing one timeout to serve both badly.

Set the startup probe's failure threshold with real margin above the observed slowest startup, not the average. A cold cache warm, a slow DNS resolution, or a dependency that's momentarily under its own load can all push a normally-20-second startup to 35 seconds on a bad day — size for the bad day, because the bad day is exactly when you can't afford a crash loop on top of whatever else is going wrong.

The exact arithmetic, because "generous timeout" is not a number. A startupProbe's total allowed startup time is periodSeconds × failureThreshold — Kubernetes probes every periodSeconds, and only gives up after failureThreshold consecutive failures. For a service whose slowest observed real-world startup is 35 seconds, periodSeconds: 5, failureThreshold: 10 allows 50 seconds total before the probe gives up and the pod is killed — real margin above the observed worst case, not the 20-second happy path. The liveness probe that takes over once startup passes can then run its own, much tighter cadence — periodSeconds: 10, failureThreshold: 3 (30 seconds to detect and act on a genuine deadlock) — because it's no longer carrying any responsibility for startup timing at all. Getting this wrong in the other direction (a startup probe with too few retries at too short a period) reintroduces the exact crash loop the probe exists to prevent, just with an extra layer of indirection that makes the on-call engineer's first guess — "the liveness probe is too aggressive" — wrong, and the actual cause one config field away from where they're looking.

A rollout is a race between old and new pods, not a switch

A rolling update keeps some old pods running while new ones start, controlled by maxUnavailable and maxSurge — it is never an instant cutover. If new pods are marked "ready" before they've actually finished warming up (because readiness was configured too loosely, or omitted entirely), the rollout will route real traffic to a not-actually-ready pod the moment it's created, causing a wave of errors during every single deploy even though nothing about the new code is broken.

Readiness is the actual gate that makes a rollout safe, not the deployment strategy's percentages. maxUnavailable: 25% controls how many pods can be down at once; it says nothing about whether a newly created pod is actually fit to serve traffic. Get readiness wrong and even maxUnavailable: 0 (theoretically "always fully available") will still route traffic to unready new pods, because the rollout controller has no other signal to know they aren't ready yet.

A pod that passes readiness once doesn't stay ready forever — readiness is checked continuously, not just at rollout time. That lets Kubernetes drain traffic from a pod that degrades hours after a deploy (stuck event loop, exhausted local resources) with no rollout in progress. Keep continuous readiness about this pod's own ability to serve — not about shared dependencies (next paragraph).

A production pattern worth naming specifically: readiness passing before the connection pool is actually warm. A common, genuinely subtle bug — reported independently by multiple engineering teams in postmortems across different companies — is a readiness probe that only checks "is the HTTP server bound and listening," which succeeds within milliseconds, while the service's database connection pool is still lazily establishing its first connections in the background. The pod reports ready, the rollout controller routes production traffic to it immediately, and the first several requests either queue behind connection establishment (a latency spike concentrated in exactly the seconds right after every deploy) or fail outright if the pool's acquire-timeout is shorter than a cold TLS handshake to the database takes under load. The fix is to gate on warm-up, once: report ready only after the pool has established its connections (and caches have loaded), using the startupProbe or a readiness check that latches "warm" and stays that way.

:::warn Never make continuous probes depend on a shared downstream If every replica's readiness runs a live database query, a brief database blip marks every replica NotReady at the same moment. The Service's endpoints go empty and a partial degradation becomes a total outage — and in-progress rollouts stall too. Liveness checking a dependency is worse: the blip triggers restarts across the whole fleet. Handle a degraded dependency with timeouts, circuit breakers and fast failures, and keep probes about the pod itself. :::

Graceful shutdown: the other half of every rollout

Most 5xx spikes during a "safe" rolling update happen when old pods stop, not when new ones start. Termination is a sequence, and two parts of it race each other:

1. The pod is marked Terminating. In parallel, the kubelet sends SIGTERM to the containers and the control plane removes the pod from the Service's EndpointSlices.
2. Endpoint removal has to propagate to every node's kube-proxy (or every load balancer and mesh proxy) — which takes a moment.
3. If the app exits on SIGTERM immediately, traffic still routed to it in that window gets connection-refused errors.
4. After terminationGracePeriodSeconds (default 30s), anything still running gets SIGKILL.

The standard fixes, in order:

• A preStop delay. Make the container wait a few seconds before SIGTERM reaches the app, so endpoint removal propagates first. Recent Kubernetes versions support a native preStop: sleep action; older setups use sleep 5 in an exec hook.
• Drain on SIGTERM. Stop accepting new connections, finish in-flight requests, close keep-alive connections, then exit — all inside the grace period. Set terminationGracePeriodSeconds above your longest realistic request plus the preStop delay.
• Make sure SIGTERM actually reaches your process. In a shell-form CMD (CMD node server.js), /bin/sh is PID 1 and doesn't forward signals, so the app never hears SIGTERM and gets SIGKILLed 30 seconds later. Use exec form (CMD ["node", "server.js"]), or an init such as tini / docker run --init.

Protect availability during voluntary disruptions too. A PodDisruptionBudget (minAvailable or maxUnavailable) stops node drains and cluster upgrades from evicting too many replicas at once — a rolling update's own maxUnavailable doesn't cover those. On the Deployment, minReadySeconds makes a new pod stay Ready for a while before it counts as available, and progressDeadlineSeconds marks a rollout as failed when it stops making progress, so automation can roll it back.

How to say it in the interview

"Liveness and readiness answer two completely different questions, and using one probe for both means you cannot ever get the timing right for both questions at once. Liveness asks 'should this be killed and restarted' - readiness asks 'should this receive traffic right now.' A slow-starting service needs the second answer to be temporarily no without the first answer ever being yes."

Checklist

• Used separate liveness and readiness probes, sized by periodSeconds × failureThreshold
• Added a startupProbe for any service with meaningfully slow startup
• Sized the startup probe's failure threshold for the slowest realistic startup, not the average
• Understood that readiness, not the rollout percentage, is what actually gates traffic to new pods
• Recognized readiness is checked continuously, not only during a rollout
• Sized startupProbe timing as periodSeconds x failureThreshold against the observed worst-case startup, not the happy path
• Gated readiness on warm-up once, and kept continuous probes off shared dependencies
• Designed graceful shutdown: preStop delay, drain on SIGTERM, exec-form CMD, and a PodDisruptionBudget

Related lessons

• Resource Requests and Limits: Sizing a Pod That Survives Real Traffic