Pages

Topics

Tools

↑↓navigate↵openescclose
All guides

Kubernetes Probes: Liveness, Readiness, and Startup Probe Configuration

Readiness removes traffic, liveness restarts the container, startup buys time for slow boots. Why a liveness probe on the database restarts every replica, and the timeout default that causes random restarts.

~6 min readKubernetes
#Kubernetes#Pods#Reliability
3D illustration of a pod cube with startup, liveness and readiness probe gauges attached

Quick Summary

Kubernetes has three probes and they answer three different questions. The readiness probe asks "should this pod receive traffic right now?" — failing it removes the pod from the Service's endpoints without restarting anything. The liveness probe asks "is this container still alive in a useful sense?" — failing it makes the kubelet kill and restart the container. The startup probe asks "has this container finished starting?" — until it succeeds, liveness and readiness are not evaluated at all, which is exactly what a slow-starting JVM or a large cache warm-up needs.

The timing fields decide everything about how a probe behaves: periodSeconds (default 10) is the check interval, timeoutSeconds (default 1) is the per-check limit, failureThreshold (default 3) is how many consecutive failures count as a failure, and successThreshold (default 1) is how many consecutive successes it takes to recover (and must be 1 for liveness and startup). A startup probe's tolerance is failureThreshold × periodSeconds, so failureThreshold: 30 with periodSeconds: 5 gives an app 150 seconds to come up before the container is killed for failing to start.

The single most common design error is pointing liveness at an endpoint that depends on a database: a short database blip then restarts every replica at once. This guide covers the handler types and their success conditions, correct timing, the separation between liveness and readiness, gRPC probes, and the failure messages that tell you which probe fired.

Basic Syntax & Core Signatures

yaml
containers: - name: api image: ghcr.io/acme/api:1.4.2 ports: [{ name: http, containerPort: 8080 }] startupProbe: httpGet: { path: /healthz, port: http } periodSeconds: 5 failureThreshold: 30 # up to 150s to start livenessProbe: httpGet: { path: /healthz, port: http } periodSeconds: 10 timeoutSeconds: 2 failureThreshold: 3 # restart after ~30s of failures readinessProbe: httpGet: { path: /readyz, port: http } periodSeconds: 5 timeoutSeconds: 2 successThreshold: 1 failureThreshold: 3 # out of endpoints after ~15s
yaml
# Handlers readinessProbe: tcpSocket: { port: 5432 } # connection succeeds livenessProbe: exec: { command: ["/bin/sh", "-c", "test -f /tmp/healthy"] } # exit 0 readinessProbe: httpGet: path: /health port: 8443 scheme: HTTPS # 200-399 is success httpHeaders: [{ name: Host, value: internal.example.com }] startupProbe: grpc: { port: 9090, service: "grpc.health.v1.Health" } # SERVING
bash
kubectl get pods -o wide # READY 1/1, 0/1 kubectl describe pod api-1 | grep -iE 'Liveness|Readiness|Startup|Unhealthy' kubectl get pod api-1 -o jsonpath='{.status.containerStatuses[0].ready}{"\n"}' kubectl get pod api-1 -o jsonpath='{.status.conditions[?(@.type=="Ready")].status}{"\n"}' kubectl get endpointslices -l kubernetes.io/service-name=api # who receives traffic
FieldDefaultApplies to
initialDelaySeconds0All probes (unnecessary with a startup probe)
periodSeconds10All probes
timeoutSeconds1All probes
successThreshold1Readiness may be > 1; must be 1 for liveness/startup
failureThreshold3All probes
terminationGracePeriodSeconds30 (pod)Restart after a failed liveness

Key Comparison Table

ProbeOn failureOn successDepends on
startupProbeContainer killed and restartedLiveness and readiness beginApp process only
livenessProbeContainer killed and restartedNothing (stays alive)App process only, never dependencies
readinessProbePod removed from endpointsPod added to endpointsApp plus its dependencies
HandlerSuccess conditionCostNotes
httpGetStatus 200-399One HTTP requestMost common; uses kubelet to pod IP
tcpSocketConnection acceptedOne TCP handshakeProves a listener, not that it serves
execExit code 0Forks a process in the containerNo shell in distroless images; CPU cost
grpcStatus SERVINGgRPC health callNeeds the standard health service implemented
SymptomWhat it means
Liveness probe failed: HTTP probe failed with statuscode: 503App said 503; the endpoint it points at checks something unhealthy
Readiness probe failed: Get "http://10.244.1.9:8080/readyz": context deadline exceededProbe timed out (default 1s)
Startup probe failed: connection refusedNothing listening yet on that port
Readiness probe errored: ... no such hostProbe host resolution failed

Practical Edge Cases & Deep Dives

1. Liveness that checks the database restarts the whole fleet

yaml
livenessProbe: httpGet: { path: /healthz, port: http } # /healthz pings the database periodSeconds: 10 failureThreshold: 3

A 40-second database failover makes every replica fail liveness at the same moment; the kubelet kills them all, they come back while the database is still down, and the loop repeats. The pod was never unhealthy — it was waiting. Liveness must test only the process's ability to make progress (not deadlocked, event loop responsive, worker threads alive). Dependencies belong in the readiness probe, where failure means "stop sending me traffic" instead of "die".

yaml
livenessProbe: { httpGet: { path: /livez, port: http } } # process only readinessProbe: { httpGet: { path: /readyz, port: http } } # dependencies included

2. The 1-second timeout that causes random restarts

timeoutSeconds defaults to 1. A Go or Node service handling a garbage collection pause, or one whose probe endpoint takes a lock, exceeds it occasionally; with failureThreshold: 3 that is three unlucky checks in 30 seconds, which a busy pod will eventually produce. The fix is a cheap probe endpoint plus a timeout that reflects real latency:

yaml
livenessProbe: httpGet: { path: /livez, port: http } timeoutSeconds: 3 periodSeconds: 10 failureThreshold: 5

The opposite mistake is a probe endpoint that performs real work (queries, writes, full health aggregation). With periodSeconds: 5 and dozens of replicas, that is load the application did not need, sometimes more expensive than the traffic the probe protects.

3. A readiness probe that never fails

yaml
readinessProbe: httpGet: { path: /healthz, port: http } # returns 200 as soon as the port opens

If readiness passes before the app can serve — connection pool empty, migrations still running, cache cold — the Deployment reports a successful rollout while clients get 500s. Readiness should include the things that make a request fail: database connectivity, required downstream clients, and any warm-up the app performs. It should not include dependencies that are optional or that would cause permanent removal from the load balancer.

The counterpart: a readiness probe so strict that a transient dependency blip removes every pod at once, which is the same failure as a liveness probe with a low threshold, just gentler.

4. Startup probes for slow starts (and what they replace)

yaml
startupProbe: httpGet: { path: /healthz, port: http } periodSeconds: 5 failureThreshold: 60 # 300 seconds

Before startup probes, slow starters needed a large initialDelaySeconds on liveness, which delayed detection of a real hang by the same amount for the whole life of the pod. A startup probe gives the container a generous budget once, then liveness runs with a tight threshold. While the startup probe is pending, readiness also does not run, so the pod is not in the endpoints — which is correct.

Note that a failing startup probe restarts the container, and repeated restarts produce CrashLoopBackOff with the startup message in the events, not a liveness message.

5. Probes, ports, and the container that listens on the wrong address

yaml
ports: [{ name: http, containerPort: 8080 }] readinessProbe: httpGet: { path: /healthz, port: http } # resolves the named port

Named ports keep probes tied to the container's declaration, so a containerPort change does not silently leave a numeric probe pointing at nothing. Two address-level problems recur: an application bound to 127.0.0.1 that the kubelet (probing the pod IP) cannot reach, and a probe against a TLS listener without scheme: HTTPS, which produces a plain-HTTP request the server rejects — visible as a 400 in the app's log and a Readiness probe failed: statuscode: 400 in the pod's events.

bash
kubectl exec deploy/api -- ss -tlnp # 0.0.0.0:8080 vs 127.0.0.1:8080

6. gRPC, exec, and the probes that cost CPU

yaml
readinessProbe: grpc: { port: 9090 } # requires the standard grpc health service periodSeconds: 10

The grpc handler calls grpc.health.v1.Health/Check and treats SERVING as success; without that service implemented, every probe fails and the pod never becomes ready. For images without a shell (distroless, scratch), exec probes cannot use sh -c or curl; the alternatives are a static health binary baked into the image or an HTTP probe against a port the app exposes.

exec probes fork a process per check, which on a CPU-limited pod competes with the application, and with periodSeconds: 5 across many replicas the cost is measurable. HTTP probes are handled by the kubelet over the pod network and cost the container only the request handling.

Common Runtime & Configuration Mistakes

SymptomCauseFix
All replicas restart at once during a dependency outageLiveness checks a dependencyMove the check to readiness
Random restarts under load1s timeout, GC pausesRaise timeoutSeconds/failureThreshold, cheap endpoint
Rollout "succeeds" but users see 500sReadiness passes before the app can serveInclude dependencies and warm-up in readiness
Pod ready immediately, traffic too earlyinitialDelaySeconds 0 with no startup probeStartup probe, or readiness that reflects state
CrashLoopBackOff on a slow-starting appStartup budget too smallRaise failureThreshold × periodSeconds
connection refused on startup probeApp not listening yet on that portVerify the port and bind address
Probe returns 400Probe uses HTTP against a TLS listenerscheme: HTTPS
exec probe fails on distrolessNo shell in the imageHTTP probe or a static binary
Probe passes but the pod gets no trafficService selector, not the probeCheck endpointslices
Pod never becomes ready with a gRPC probeHealth service not implementedImplement grpc.health.v1
Probe traffic dominates logsperiodSeconds too small10-30s is normal
Readiness flaps, endpoints churnThresholds too tight for normal latencyfailureThreshold: 3, successThreshold: 1-2

Probes are a contract between the workload and the platform: readiness decides traffic, liveness decides restarts, and startup decides how long the platform should wait before judging either. Set liveness to test only the process, readiness to test what actually serves requests, and a startup probe generous enough for the slowest real boot — and read the event line naming the probe when something fails, because it says which contract was broken.

Try matching browser tools

All tools

Small utilities that fit Kubernetes work — they run entirely in this browser tab, so nothing you paste leaves your device.

All Kubernetes guides

More in Kubernetes

All Kubernetes guides

The guides filed next to this one in Kubernetes, in library order.