Table of Contents
By Khimananda Oli | Last reviewed: August 2026
Misconfigured health check endpoints are a leading cause of phantom outages and cascading failures in Kubernetes clusters. While both probes signal application status, confusing health check endpoints: liveness vs readiness leads to restart loops when dependencies lag or traffic routing to unprepared pods. This guide clarifies the distinct operational roles of each probe so you can configure them correctly for production workloads.
What is the fundamental difference between liveness and readiness probes?
The distinction lies entirely in the remediation action the orchestrator takes upon failure. Understanding this prevents the most common class of probe-related incidents I see in production audits.
Liveness: Should this process exist?
A liveness probe answers one question: "Is the application process capable of making forward progress?" If the answer is no, Kubernetes kills the container and restarts it according to the pod's restart policy. This is appropriate for deadlocked threads, corrupted in-memory state, or infinite loops where only a fresh process can recover functionality. Critically, a liveness failure implies the current instance is unfixable without restart.
Readiness: Can this instance serve traffic right now?
A readiness probe answers: "Is this specific pod ready to accept and successfully handle requests?" Failure here removes the pod from service endpoints but leaves the container running. This handles slow startups, temporary dependency outages (database failover, cache warming), or maintenance modes. The key insight: readiness failures are often transient and external to the application process itself. For teams implementing meaningful SLIs and SLOs, readiness directly maps to availability metrics.
How do you configure health check endpoints correctly in Kubernetes?
Correct configuration requires matching probe type to failure mode. Here is a production-grade pattern for a typical web API with database dependencies.
apiVersion: apps/v1
kind: Deployment
metadata:
name: api-service
spec:
template:
spec:
containers:
- name: api
image: myregistry/api:v2.4.1
ports:
- containerPort: 8080
# LIVENESS: Lightweight, internal-only check
livenessProbe:
httpGet:
path: /healthz/live
port: 8080
initialDelaySeconds: 15
periodSeconds: 20
timeoutSeconds: 3
failureThreshold: 3
# READINESS: Validates full request path viability
readinessProbe:
httpGet:
path: /healthz/ready
port: 8080
initialDelaySeconds: 5
periodSeconds: 10
timeoutSeconds: 5
failureThreshold: 2
# STARTUP: Prevents premature liveness kills during boot
startupProbe:
httpGet:
path: /healthz/startup
port: 8080
failureThreshold: 30
periodSeconds: 2Three implementation rules emerge from real-world debugging sessions:
- Never share endpoints. A single
/healthendpoint that checks both process state and dependencies will either cause unnecessary restarts (if used as liveness) or mask true process death (if used as readiness). - Use startup probes for slow-initializing apps. Without a startup probe, aggressive liveness settings kill pods before they finish loading models, warming caches, or running migrations. The startup probe disables liveness until the app signals basic viability.
- Set timeouts shorter than periods. A timeout exceeding the period creates overlapping checks that compound load during degradation. Keep timeouts at 30–50% of the period interval.
Application-side endpoint design
Your application must expose semantically correct endpoints. The liveness handler should return 200 if the HTTP server is responsive and core goroutines/threads are active—no database calls, no external HTTP requests. The readiness handler validates the full dependency chain required to serve user traffic. When integrating Prometheus metrics monitoring, expose probe latency as histograms to detect degrading health check performance before it causes timeouts.
Why does using readiness logic in liveness probes cause restart storms?
This is the single most damaging misconfiguration pattern. When your liveness probe checks database connectivity and the database experiences a 30-second failover, every pod fails its liveness check simultaneously. Kubernetes restarts all pods concurrently, creating a thundering herd against the recovering database, which extends the outage exponentially.
I diagnosed this exact scenario for a fintech client in Kathmandu last year. Their payment service used a shared /health endpoint for both probes. During routine RDS maintenance, all 12 pods entered CrashLoopBackOff because liveness failures triggered mass restarts precisely when the database needed reduced load. The fix was separating concerns: liveness checked only the HTTP listener, readiness validated the DB connection pool. Recovery time dropped from 8 minutes to under 45 seconds.
The principle: liveness scope must be strictly narrower than readiness scope. If a condition is survivable without restart (temporary network blip, dependency timeout, queue backlog), it belongs in readiness only. Reserve liveness for conditions where continued execution guarantees incorrect behavior or resource exhaustion. Teams practicing blue-green and canary deploys find this separation especially critical during progressive rollouts where partial dependency availability is expected.
How do probe parameters affect reliability and recovery time?
Parameter tuning determines whether your health checks provide graceful degradation or amplify failures. Default values rarely suit production workloads.
| Parameter | Liveness Recommendation | Readiness Recommendation | Rationale |
|---|---|---|---|
initialDelaySeconds | Minimal (use startup probe instead) | Low (5–10s) | Readiness should gate traffic immediately post-startup; liveness relies on startup probe completion |
periodSeconds | 15–30s | 5–15s | Readiness needs faster reaction to dependency changes; liveness tolerates slower detection |
timeoutSeconds | 2–5s | 3–8s | Readiness may legitimately wait longer for dependency responses; liveness timeouts must be short to detect hangs |
failureThreshold | 3–5 | 2–3 | Higher liveness threshold prevents restarts from transient GC pauses; lower readiness threshold stops bad traffic faster |
successThreshold | Always 1 (fixed) | 1–3 | Requiring consecutive successes for readiness prevents flapping during dependency recovery |
A common mistake: setting identical parameters for both probes. Readiness should react faster to failures (lower threshold, shorter period) because serving traffic to an unhealthy pod has immediate user impact. Liveness can afford more tolerance because restarts are expensive and disruptive. When tuning these values, correlate them with your four golden signals baselines—probe sensitivity should align with your actual error rate and latency SLOs.
TCP and exec probes: when HTTP isn't appropriate
Not every workload speaks HTTP. For message queue consumers, batch processors, or legacy binaries, use TCP socket checks (port open = alive) or exec probes (custom script exit code). However, exec probes carry overhead: each invocation spawns a new process inside the container. At high frequencies, this consumes CPU and can interfere with cgroup limits. Prefer TCP over exec when possible, and never run complex validation scripts in liveness exec probes.
Implementing resilient health check endpoints for production
Getting health check endpoints: liveness vs readiness right requires treating them as first-class API contracts, not afterthoughts. Apply these practices consistently across your fleet.
- Version your health endpoints. Use
/healthz/v1/liverather than/healthz/live. When you need to change probe semantics during a migration, versioned paths let old and new deployments coexist without breaking existing probe configurations. - Add structured logging to readiness failures. When readiness fails, log which dependency failed and why. Silent 503s make debugging impossible during incidents. Include correlation IDs so probe failures correlate with downstream logs.
- Expose probe metrics. Track probe response latency and status codes as Prometheus metrics. Degraded probe performance often precedes application issues. Alert on p99 probe latency exceeding 50% of your configured timeout.
- Test probes in CI. Include integration tests that verify liveness returns 200 without dependencies, and readiness returns 503 when mocked dependencies fail. Catch semantic errors before deployment.
- Document probe contracts. Maintain runbooks explaining what each endpoint checks and why. On-call engineers need this context at 3 AM when deciding whether to adjust thresholds or escalate.
For teams managing databases alongside application health, understanding PostgreSQL administration essentials helps design readiness checks that distinguish between acceptable replication lag and genuine connection failures.
Next steps for reliable probe configuration
Correctly implemented health check endpoints eliminate an entire category of self-inflicted outages. Audit your current deployments: separate any shared endpoints, add startup probes to slow-starting services, and tune parameters based on actual dependency behavior rather than defaults. Monitor probe metrics alongside your application SLOs to validate that your configuration matches reality. If your team needs help designing probe strategies that align with compliance requirements or multi-cloud architectures, reach out to discuss your specific infrastructure challenges.
