Readiness and Liveness Probes in Kubernetes: Why a Healthy Pod Can Still Serve Broken Traffic

A rolling deploy finishes, kubectl get pods shows everything Running, the dashboard is green, and yet for a few seconds some clients get 502 or connection refused. The diagnosis is almost always the same: the pod was alive, but it wasn't ready when the Service already started routing traffic to it. That's the gap between two questions Kubernetes can answer separately, and that almost no default manifest tells apart.

Three probes, three different questions

Kubernetes doesn't have one health check, it has three, and each answers something different:

• liveness: is the process still working, or does it need to be killed and restarted? If it fails, the kubelet restarts the container.
• readiness: can this pod take traffic right now? If it fails, the pod is pulled from the Service's endpoints, but nothing restarts.
• startup: has the app finished starting up? Until this passes, liveness and readiness aren't evaluated at all.

The most common mistake we see in existing manifests isn't skipping probes altogether, it's shipping only a livenessProbe and assuming it also protects incoming traffic. It doesn't. Without a readinessProbe, Kubernetes treats a container as ready the moment it reaches Running, which is a signal from the container runtime, not from the application. A Java process can be "Running" while the JVM is still loading classes and the database connection pool hasn't even been initialized. During that window, if the Service already lists it as an endpoint, any request that lands on it fails.

The typical case: slow startup on the JVM and .NET

This shows up most on runtimes with non-trivial startup. A typical Spring Boot app opens its HTTP port before every @Bean has finished initializing, unless the startup order was deliberately designed around that. A .NET service on Kestrel can accept TCP connections on its configured port before the IHostedService that warms the cache or opens the connection pool has finished running. In both cases, the port is open — the pod looks ready from the outside — but the application can't actually handle a real request yet without failing or hanging.

# Incomplete config: without a readinessProbe, the Service routes
# traffic as soon as the port answers, not when the app is actually ready.
containers:
  - name: api
    image: registry/my-api:1.4.0
    ports:
      - containerPort: 8080
    livenessProbe:
      httpGet:
        path: /health
        port: 8080

Nothing here is technically wrong. It's just missing the right question.

Why the Kubernetes defaults don't cover it either

Once a readinessProbe is added, the second problem is leaving Kubernetes' default probe values untouched. Those defaults are initialDelaySeconds: 0, periodSeconds: 10, timeoutSeconds: 1, successThreshold: 1, and failureThreshold: 3. That means the first check runs the instant the container starts, and after three consecutive failures — three attempts, thirty seconds — the kubelet is already acting on it.

For a readinessProbe alone, thirty seconds of failures just keeps the pod out of the Service, which is the correct outcome: no traffic beats broken traffic. The real problem shows up when those same defaults get copied onto the livenessProbe of a slow-starting service. If the app takes forty seconds to actually be operational, and the livenessProbe starts failing against the same endpoint from second zero, the kubelet restarts the container at the thirty-second mark. It boots again, takes another forty seconds, fails the probe again, restarts again. That's a CrashLoopBackOff with no application bug behind it: it's a miscalibrated probe killing a process that just needed more time.

startupProbe: the missing piece

The fix isn't stretching the livenessProbe's initialDelaySeconds to some generous number and hoping for the best. That works until someone changes the typical startup time and nobody revisits that value. The fix is startupProbe, which exists for exactly this case: as long as the startupProbe hasn't succeeded, neither livenessProbe nor readinessProbe gets evaluated. Once the startupProbe passes, it stops running and the other two take over with their own thresholds — thresholds that can now be strict, since they no longer have to cover the startup window.

containers:
  - name: api
    image: registry/my-api:1.4.0
    ports:
      - containerPort: 8080
    startupProbe:
      httpGet:
        path: /health/startup
        port: 8080
      periodSeconds: 5
      failureThreshold: 24        # up to 120s of margin to start up
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 8080
      periodSeconds: 5
      failureThreshold: 2         # 10s out of traffic if something degrades
    livenessProbe:
      httpGet:
        path: /health/live
        port: 8080
      periodSeconds: 10
      failureThreshold: 3         # strict: startup is already done

With this setup, a slow startup gets up to 120 seconds to complete without anything killing it, but once it's up, the livenessProbe reacts within 30 seconds to a genuinely hung process, and the readinessProbe pulls the pod out of traffic within 10 seconds if a dependency starts failing.

Sizing the thresholds instead of guessing

The failureThreshold for the startupProbe doesn't come from a generic table. It comes from measuring that specific application's real startup under load comparable to production, startup logs in hand, with a reasonable margin over the worst case observed — not the typical case. A service that starts in 15 seconds on a developer's laptop and in 45 seconds on the cluster under the pod's actual CPU limits is exactly why that margin matters: a low resources.requests.cpu on a container with a heavy startup path stretches initialization time, because the kubelet throttles how much CPU the process can use, and the same binary ends up taking a different amount of time depending on the cluster it lands on.

The mistake of pointing both probes at the same endpoint

The second anti-pattern, beyond skipping the startupProbe, is pointing readinessProbe and livenessProbe at the same endpoint running the same internal checks. If that endpoint verifies the database connection and the database has an outage, both probes fail at once: the pod drops out of the Service (correct) and the kubelet starts restarting the container (wrong, because restarting the process doesn't fix the database outage). The result is a set of pods stuck in a restart loop that doesn't solve the actual problem, and that adds log noise that makes the root cause harder to find.

The right split is for livenessProbe to check only that the internal process is responsive — without touching external dependencies — and for readinessProbe to check the dependencies the app actually needs to serve traffic. A process can be perfectly alive and still not be ready because its database isn't responding; that pod doesn't need a restart, it needs to come out of the Service until the dependency comes back.

What this doesn't fix

Tuning the three probes correctly stops a pod from taking traffic before it's ready, and stops it from restart-looping over a slow startup, but it doesn't cover the other end of the lifecycle: shutdown. When Kubernetes decides to terminate a pod, it can keep sending it traffic for a brief window while the endpoint update propagates, and if the container doesn't handle SIGTERM with a preStop hook or a terminationGracePeriodSeconds long enough to drain in-flight connections, that window produces the same kind of error we were trying to avoid with the startup probes, just at the opposite end. It's a related but separate problem, and it's worth handling on its own instead of trying to patch it through readinessProbe too.