Liveness and readiness in Kubernetes: when the probe kills more pods than it saves

The first time we configure probes on a deployment, the temptation is almost always the same: copy an example from the docs, point livenessProbe and readinessProbe at the same health endpoint, tune a couple of timers, and call it done. It works fine in development, where the service is never under real pressure. It fails in production in a very specific way: under load, the same pod that was handling requests normally starts responding more slowly, the liveness probe reads that slowness as a failure, and the kubelet restarts a container that was never down, just busy. The restart fixes nothing, because nothing was broken; it just drops the work in flight, flushes in-memory caches, and removes one pod's worth of capacity at the exact moment more capacity was needed.

Two different questions answered by the same endpoint

Liveness and readiness answer different questions, even though the natural instinct is to treat them as one. The liveness probe answers "is this process still alive and should it keep running, or is it in a state it will never recover from on its own?" The readiness probe answers "is this pod in a condition to receive traffic right now?" When the answer to the first is no, there is exactly one fix: restart the container. When the answer to the second is no, there is a much cheaper fix: pull it out of the Service's endpoints temporarily until it recovers, without touching the process at all.

The mistake we see most often is a liveness probe doing the same job as readiness: hitting an endpoint that depends on the database, a queue, or any external service.

# Configuration that punishes slowness as if it were a hang
livenessProbe:
  httpGet:
    path: /health
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 1
  failureThreshold: 2

If /health queries the database and the database is answering slowly because of a legitimate traffic spike, the probe fails twice in ten seconds and the kubelet restarts the container. The application process never stopped responding — it just took longer than the probe was willing to wait. The restart does nothing about the database's slowness, removes a pod that was actually helping absorb the load, and during a high-traffic event it can trigger a loop where each restart shrinks capacity, which raises load on the remaining pods, which makes their own liveness probes start failing too.

What each probe should actually check

The rule we use to keep the two responsibilities apart is simple to state and easy to forget under delivery pressure: the liveness probe should only confirm that the process itself is not in an unrecoverable state — a deadlock, an infinite loop, corrupted memory the process can't clean up on its own — and it should never depend on a network call to an external service. The readiness probe can, and should, check those dependencies, because deciding whether the pod can serve traffic given the current state of its dependencies is exactly its job.

livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  periodSeconds: 10
  timeoutSeconds: 2
  failureThreshold: 3
readinessProbe:
  httpGet:
    path: /healthz/ready
    port: 8080
  periodSeconds: 5
  timeoutSeconds: 2
  failureThreshold: 3

/healthz/live returns 200 as long as the process can accept and dispatch a basic HTTP request, without touching the database or any external dependency. /healthz/ready does query those dependencies, and when one of them isn't responding, the pod drops out of the Service's endpoints without the kubelet ever touching it. If the database recovers thirty seconds later, the next readiness check marks the pod ready again and traffic flows back on its own — without ever losing the process, its warm connections, or whatever it had in memory.

When a slow startup looks like being down

A second pattern that produces the same symptom — healthy pods restarted for no reason — shows up during startup rather than during normal operation. A service that needs time to warm a cache, load a model into memory, or wait on a schema migration can take longer to become ready than the liveness probe is willing to tolerate before its first check, and it ends up stuck in a restart loop where it never finishes starting because every attempt gets interrupted before completion.

startupProbe exists specifically for this case: while the startup probe hasn't reported success, neither the liveness nor the readiness probe runs at all, so the startup process gets whatever time it needs without competing against thresholds meant for steady-state operation.

startupProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  periodSeconds: 5
  failureThreshold: 30
livenessProbe:
  httpGet:
    path: /healthz/live
    port: 8080
  periodSeconds: 10
  failureThreshold: 3

With periodSeconds: 5 and failureThreshold: 30, the pod gets up to two and a half minutes to finish starting before Kubernetes gives up on it. Past that point, control shifts to the regular liveness probe, which can now use tighter timing because the process is running in steady state, not booting.

What didn't work: lowering failureThreshold to "react faster"

On a service with occasional latency spikes under load, our first response to seeing healthy pods get restarted wasn't to split liveness from readiness — it was to lower the liveness probe's failureThreshold from three to one, on the theory that a faster restart on a real problem would limit the blast radius better. The result was the opposite. With only one failure tolerated, any transient latency blip — a runtime garbage collection pause, a second of CPU contention on a shared node — was enough to trigger a restart. Restarts got more frequent, not less, and each one cost more than the original slowness it was meant to prevent, because a freshly restarted container takes time to re-warm its own connections and caches, opening a new window of slowness right after the restart that was supposed to eliminate one.

We reverted that change and, instead, moved the external-dependency check from the liveness probe to the readiness probe, which is actually built to tolerate and communicate that kind of degradation without killing the process. Tightening thresholds on the wrong probe doesn't fix a design problem — it just changes how fast the problem shows up.

What we check before signing off on a probe configuration

Before we consider a deployment's probe configuration finished, we verify that the liveness probe has no network dependency on any external service, that the readiness probe covers those dependencies and is the only one responsible for pulling a pod out of load balancing, that a startup probe exists whenever startup time can exceed the liveness probe's periodSeconds × failureThreshold, and that the liveness thresholds are generous enough to absorb transient latency spikes without losing the ability to catch a process that's genuinely hung. Once those four hold, a traffic spike stops being an event that Kubernetes makes worse on its own, and goes back to being exactly what it should be: a signal to scale out, not a reason to restart something that was already working.