Table of Contents
By Khimananda Oli | Last reviewed: August 2026
When a production service stops responding at 2 AM, generic documentation rarely solves the immediate problem. Effective Kubernetes Troubleshooting: A Field Guide requires a structured methodology that moves from symptom identification to root cause analysis without guessing. This guide distills years of operational experience into actionable diagnostic workflows for developers and platform engineers managing live clusters. Before diving into complex log aggregation or distributed tracing, you must master the fundamental kubectl inspection patterns outlined in our Kubernetes basics deployment guide to establish a reliable baseline.
kubectl get pods -o wide, check describe output for scheduling or image errors, then examine container logs and node conditions to isolate application versus infrastructure failures accurately.How do you systematically diagnose failing Kubernetes pods?
The most common failure point in any cluster is the pod lifecycle. When workloads fail to reach a Ready state, you need a repeatable inspection sequence rather than random command execution. Start by widening your view beyond default columns to see node placement, restart counts, and IP assignments simultaneously.
kubectl get pods -n production -o wide --sort-by=.metadata.creationTimestampThis output immediately reveals patterns: are all failures on a single node? Is there a cascade of restarts indicating an OOMKill loop? Are pods stuck in ContainerCreating across multiple nodes, suggesting a CNI or image pull issue? Once you identify the specific failing pod, the describe command provides the authoritative event timeline.
Interpreting Events and Conditions
The Events section in kubectl describe pod output is ordered chronologically but often truncated. Always check the full event history with timestamps to correlate failures with deployments or node maintenance windows. Pay special attention to Warning events like FailedScheduling, which indicate resource constraints, or FailedMount, which points to persistent volume attachment problems. For deeper context on storage-related failures, review Kubernetes persistent volumes and storage configuration patterns.
- ImagePullBackOff: Verify image tag existence, registry credentials, and network egress rules. Test pulls manually from the affected node using crictl or docker.
- CreateContainerConfigError: Usually indicates missing Secrets or ConfigMaps referenced in the pod spec. Validate references exist in the correct namespace.
- OOMKilled: Container exceeded memory limits. Check actual usage via metrics-server or Prometheus before blindly increasing limits. See Kubernetes resource limits and requests for right-sizing guidance.
- CrashLoopBackOff: Application exits non-zero repeatedly. Examine previous container logs with
--previousflag to capture the fatal error message.
What causes CrashLoopBackOff and how do you fix it?
CrashLoopBackOff is arguably the most frequent issue covered in any Kubernetes Troubleshooting: A Field Guide. It means the container starts but terminates unexpectedly, and kubelet backs off restart attempts exponentially. The critical mistake engineers make is only looking at current logs, which may show startup messages before the crash. You must retrieve logs from the terminated instance.
# Get logs from the previously crashed container instance
kubectl logs myapp-deployment-7b9f4d6c8-xk2lp --previous -n production
# If multi-container pod, specify the container name
kubectl logs myapp-pod --previous -c app-container -n productionIf the application crashes before producing any log output, the issue is likely environmental: missing environment variables, unreadable config files, permission errors on mounted volumes, or incompatible binary architecture. Use an ephemeral debug container to inspect the runtime environment without modifying the deployment.
kubectl debug -it myapp-pod --image=busybox:1.36 --target=app-container -n productionThis attaches a temporary container sharing the target's namespaces, allowing you to verify file permissions, test DNS resolution, check environment variables, and validate network connectivity from inside the actual pod context. Remember that debug containers are transient and won't persist across restarts.
How do you troubleshoot Kubernetes networking and service connectivity?
Network issues manifest as timeouts, connection refused errors, or DNS resolution failures between services. Unlike pod crashes, these problems span multiple layers: CNI plugin health, Service/Endpoint objects, NetworkPolicies, and DNS configuration. Always verify the entire chain systematically.
Validating Service Endpoints and DNS
A Service with zero endpoints will silently drop traffic. Confirm endpoint population matches your pod selector labels exactly.
# Check if endpoints are populated
kubectl get endpoints my-service -n production -o yaml
# Verify DNS resolution from within the cluster
kubectl run dns-test --rm -it --image=busybox:1.36 --restart=Never -- nslookup my-service.production.svc.cluster.localIf DNS fails but direct IP access works, CoreDNS is misconfigured or overwhelmed. Check CoreDNS pod logs and resource utilization. For comprehensive networking policy debugging including ingress controller validation, consult Kubernetes network policies explained.
Testing Connectivity at Each Layer
Use curl or wget from a debug pod to test connectivity progressively: first to the Service ClusterIP, then to individual pod IPs, then to external dependencies. This isolates whether the failure is in kube-proxy rules, CNI routing, or application-level firewalls. Document each test result to build evidence for post-incident reviews.
| Connectivity Test | Command Pattern | Failure Indicates | Next Diagnostic Step |
|---|---|---|---|
| DNS Resolution | nslookup svc.namespace.svc | CoreDNS failure or search domain misconfiguration | Check CoreDNS logs, ConfigMap, and pod resources |
| Service ClusterIP | curl -v http://svc-ip:port/health | kube-proxy iptables/IPVS rules missing or stale | Verify endpoints exist, check kube-proxy logs on node |
| Direct Pod IP | curl -v http://pod-ip:port/health | CNI plugin failure or network policy blocking | Inspect CNI logs, validate NetworkPolicy selectors |
| External Dependency | curl -v https://api.external.com | Egress network policy or NAT gateway misconfiguration | Review egress policies, check node SNAT rules |
Why are Kubernetes nodes NotReady and how do you recover them?
Node-level failures affect all scheduled workloads and require infrastructure-focused diagnostics. A NotReady condition typically stems from kubelet crashes, container runtime failures, disk pressure, or network partitioning. SSH into the affected node and check systemd service status first.
# On the affected node
systemctl status kubelet containerd
journalctl -u kubelet --since "1 hour ago" --no-pager | tail -100
# Check for resource exhaustion
df -h /var/lib/containerd
free -m
top -bn1 | head -20Disk pressure is especially common in environments without proper log rotation or garbage collection tuning. Container runtimes accumulate unused images and stopped containers rapidly under high churn. Configure kubelet eviction thresholds proactively and implement automated cleanup routines. For clusters running database workloads where disk I/O contention compounds node instability, reference PostgreSQL administration essentials for storage-aware deployment patterns.
Recovering Stuck Nodes Safely
Before cordoning or draining a problematic node, verify no critical single-replica workloads are stranded. Use kubectl drain --ignore-daemonsets --delete-emptydir-data to gracefully evict pods while preserving DaemonSets. If the node is completely unresponsive, delete it from the API server and allow the cloud provider or provisioning tool to replace it. Never force-delete pods unless you've confirmed data persistence elsewhere.
How do you prevent recurring Kubernetes issues through observability?
Reactive troubleshooting becomes unsustainable at scale. Building proactive detection requires instrumenting the four golden signals: latency, traffic, errors, and saturation. Configure alerts on meaningful SLIs rather than raw metrics to reduce noise. Establish baseline performance profiles for normal operations so anomalies stand out clearly during incidents. Integrate structured logging practices early to ensure log data is queryable during crises rather than buried in unstructured text blobs. Review the four golden signals of monitoring to align your observability strategy with SRE best practices.
Automate repetitive diagnostics using kubectl plugins or custom scripts wrapped in CI health checks. Capture successful resolution steps in runbooks stored alongside your infrastructure code. Every incident should produce either a new alert rule, an improved dashboard, or updated documentation. This feedback loop transforms painful 2 AM debugging sessions into lasting operational improvements that compound over time.
Building Resilience Through Methodical Kubernetes Troubleshooting
Mastering Kubernetes Troubleshooting: A Field Guide principles turns chaotic incident response into predictable engineering work. The difference between junior and senior operators isn't memorizing every flag—it's maintaining disciplined investigation sequences under pressure. Build muscle memory with these workflows in staging environments before relying on them during production outages. Invest time now configuring proper resource requests, health probes, and observability pipelines so future incidents resolve faster. If your team needs hands-on support establishing production-grade cluster operations or audit-ready infrastructure, reach out to discuss your specific challenges.
