← Back

Kubernetes Probes: Readiness Is Not Liveness

Choose Kubernetes readiness, liveness, and startup probes by the action each failure should trigger, then test dependency outages and slow startup behavior.

Kubernetes readiness and liveness probes answer different operational questions. Readiness asks whether this instance should receive traffic. Liveness asks whether restarting the container is an appropriate response. A dependency outage can make the first answer “no” without making a restart useful.

My starting rule is to design the failure action before designing the endpoint. A health check is a control input to the platform, so an inaccurate failure signal can be worse than no signal.

Write the action beside the condition

Kubernetes documents separate startup, readiness, and liveness behavior. Failed readiness changes traffic eligibility; repeated liveness failure can trigger a restart. A startup probe delays liveness and readiness checks until startup succeeds. The documentation explicitly warns that badly designed liveness checks can contribute to cascading failures. Kubernetes probe concepts.

For a hypothetical HTTP service, use a review table:

ConditionDesired actionCandidate signal
Initialization still progressingWait within an allowed startup budgetStartup
Instance cannot serve its required workloadStop routing new traffic to itReadiness
Process is stuck in a state restart can repairRestart after configured failuresLiveness
Shared database unavailableEvaluate degraded service and routing policyUsually not a restart signal

The final row needs application judgment. If every replica depends on the same unavailable database, removing every replica from service does not restore the database. It may still be correct if no useful response can be served, but the resulting behavior should be intentional.

Keep liveness focused on recoverable local failure

Ask what a restart would change. A deadlocked local worker may be a useful liveness target. A remote API outage probably survives the restart. A slow request under overload may become slower if the platform repeatedly destroys warm instances.

I would avoid a liveness endpoint that performs a full tour of every dependency. That endpoint can turn a shared external fault into synchronized local disruption. Keep its work bounded and tied to conditions the application can actually recover from through restart.

A static endpoint that always returns success has the opposite problem: it can report healthy while the useful processing path is stuck. Design a cheap check of the relevant local progress or state. The right signal depends on the server architecture; there is no universally correct /health implementation.

Also distinguish startup from deadlock. Loading a large dataset or establishing an initial state may take longer than steady-state operations. An instance should not be repeatedly killed while doing expected initialization.

Make readiness reflect useful service

Define what the instance can still do during a dependency failure. A catalog may serve cached reads while checkout is unavailable. A specialized worker may have no useful work until its queue connection returns. Those services should not inherit the same readiness logic merely because they share a framework.

If readiness includes a dependency check, bound the check and decide how fresh its result must be. Avoid creating a second load generator that aggressively queries the struggling dependency from every replica. A maintained internal state may be more appropriate than synchronous probing on every health request, provided its staleness is understood.

Readiness also needs a recovery path. The instance must continue checking or otherwise learning that its condition improved. A one-way switch to unhealthy that only a restart resets turns a temporary fault into a persistent routing problem.

Tune thresholds from the behavior you need

Kubernetes exposes fields such as periodSeconds, timeoutSeconds, failureThreshold, and initialDelaySeconds; its configuration guide explains their semantics and examples. Read the documentation for your probe type and cluster version instead of treating copied values as universal defaults. Kubernetes probe configuration.

Start with observed startup and response behavior in a representative environment. Decide how long you are willing to route to an unhealthy instance and how much transient failure you want to tolerate before restarting. Those decisions guide the settings.

A short timeout can detect failure quickly but can also classify temporary scheduling delays as failure. A generous threshold reduces flapping but delays corrective action. Neither extreme is inherently reliable. Document the trade-off beside the workload rather than leaving it hidden in deployment YAML.

Treat any calculated detection time as an approximation unless you have measured the actual system. Scheduling, probe execution, and platform behavior complicate simplistic arithmetic.

Test failure scenarios before relying on the probes

In a test environment, exercise slow startup, a locally stuck process, a failed dependency, and overload. Observe the sequence: probe results, readiness state, traffic routing, restarts, and recovery.

The valuable test assertion is behavioral. “The endpoint returns 503” proves only the endpoint response. “Traffic stops reaching the unhealthy replica while healthy replicas continue serving, and the replica returns after recovery” tests the intended platform interaction.

Include a case where restarting would not help. If the service enters a restart loop during a shared outage, revise the liveness design. Include a case where readiness becomes healthy again without manual intervention.

The SLO policy helps define which service behavior readiness should protect. The connection-pool overload exercise provides a dependency-pressure scenario to test before trusting the health signal. Probes deserve the same review as application logic.