Table of Contents
By Khimananda Oli | Last reviewed: August 2026
Dropped requests during rolling updates remain a primary source of user-facing errors in microservices, even when infrastructure is correctly configured. Implementing graceful shutdown and health checks in Go requires coordinating OS signal handling, HTTP server lifecycle management, and distinct readiness versus liveness semantics. Without this coordination, load balancers continue routing traffic to terminating pods, causing intermittent 502 errors that are difficult to trace. This guide provides the exact implementation pattern I use in production to ensure zero-downtime deployments on Kubernetes.
How do you implement graceful shutdown and health checks in Go with signal handling?
The foundation of reliable Go services is proper signal propagation using context. Before implementing probes or server logic, you must establish a root context that cancels when the operating system sends a termination signal. In Go 1.21+, signal.NotifyContext simplifies this significantly compared to older channel-based patterns.
Setting up the cancellation root
Your main function should create a context that listens specifically for SIGTERM and SIGINT. Kubernetes sends SIGTERM during pod termination, while SIGINT covers local development (Ctrl+C). Never listen for SIGKILL; it cannot be caught by design.
package main
import (
"context"
"log/slog"
"os"
"os/signal"
"syscall"
)
func main() {
ctx, stop := signal.NotifyContext(
context.Background(),
os.Interrupt,
syscall.SIGTERM,
)
defer stop()
logger := slog.New(slog.NewJSONHandler(os.Stdout, nil))
// Pass ctx to all long-lived components
if err := run(ctx, logger); err != nil {
logger.Error("application error", "error", err)
os.Exit(1)
}
}This pattern ensures every goroutine, database connection, and HTTP handler receives the same cancellation signal. For teams managing complex dependencies, understanding Kubernetes resource limits and requests is essential because insufficient CPU can delay signal processing, causing hard kills before graceful shutdown completes.
Why context matters more than channels
Older tutorials often show raw channel selects for signal handling. Avoid this in 2026. Context propagation integrates naturally with Go's standard library servers, database drivers, and gRPC clients. When the context cancels, pending operations abort automatically without manual teardown logic scattered across your codebase.
What is the difference between liveness and readiness probes in Go?
Confusing these two probes is the most common cause of deployment failures I see during audits. They serve fundamentally different purposes, and conflating them creates cascading failures during restarts.
| Characteristic | Liveness Probe | Readiness Probe |
|---|---|---|
| Purpose | Detect deadlocks or unrecoverable state | Determine if pod should receive traffic |
| Failure Action | Kubelet restarts the container | Endpoints controller removes pod from Service |
| During Shutdown | Must continue returning 200 | Must return 503 immediately on SIGTERM |
| Dependency Checks | Never check external dependencies | Optionally verify critical dependencies |
| Path Convention | /health/live | /health/ready |
The shutdown-critical distinction
During graceful shutdown, your liveness probe must remain healthy until the process actually exits. If liveness fails during drain, Kubernetes kills the pod mid-request. Conversely, readiness must fail the moment SIGTERM arrives so the load balancer stops routing new traffic while existing connections complete. This asymmetry is non-negotiable for graceful shutdown and health checks in Go.
For deeper observability integration beyond basic probes, refer to the four golden signals of monitoring to understand which metrics correlate with probe failures in production dashboards.
How do you configure Kubernetes probes for Go applications?
Application code alone does not guarantee zero-downtime. Your Kubernetes manifest must align probe timing with Go's shutdown behavior. Misconfigured timeouts cause pods to be killed before in-flight requests complete, negating all application-level effort.
Recommended probe configuration
- readinessProbe.initialDelaySeconds: 2–5 (Go starts fast; don't wait 30s)
- readinessProbe.periodSeconds: 5 (detect unready state quickly during rollout)
- readinessProbe.failureThreshold: 1 (remove from LB on first failure during shutdown)
- livenessProbe.initialDelaySeconds: 10 (allow startup without premature restart)
- livenessProbe.periodSeconds: 15 (avoid excessive polling)
- livenessProbe.failureThreshold: 3 (tolerate transient GC pauses)
- terminationGracePeriodSeconds: Match your max request duration + buffer (default 30s is often too short)
The preStop hook safety net
Even with perfect readiness logic, there is a race condition between the kubelet sending SIGTERM and the endpoints controller updating iptables/IPVS rules. Adding a preStop hook with a 3–5 second sleep gives the network layer time to propagate the endpoint removal before your app stops accepting connections:
lifecycle:
preStop:
exec:
command: ["/bin/sh", "-c", "sleep 5"]This is especially critical in high-throughput Nepali e-commerce platforms where even brief windows of misrouted traffic during flash sales cause visible customer impact. Combine this with blue-green and canary deploys on Kubernetes to further reduce blast radius during releases.
How do you handle in-flight requests during Go server shutdown?
http.Server.Shutdown() stops accepting new connections and waits for active requests to complete, but it does not manage downstream resources. You must coordinate database pools, message queue consumers, and cache connections to close only after HTTP draining finishes.
Complete server implementation
func run(ctx context.Context, logger *slog.Logger) error {
mux := http.NewServeMux()
// Readiness state managed via atomic or channel
ready := make(chan struct{})
close(ready) // Initially ready
mux.HandleFunc("/health/live", func(w http.ResponseWriter, r *http.Request) {
w.WriteHeader(http.StatusOK)
w.Write([]byte("ok"))
})
mux.HandleFunc("/health/ready", func(w http.ResponseWriter, r *http.Request) {
select {
case <-ready:
w.WriteHeader(http.StatusServiceUnavailable)
w.Write([]byte("shutting down"))
default:
w.WriteHeader(http.StatusOK)
w.Write([]byte("ok"))
}
})
srv := &http.Server{
Addr: ":8080",
Handler: mux,
}
errCh := make(chan error, 1)
go func() {
logger.Info("server starting", "addr", srv.Addr)
if err := srv.ListenAndServe(); err != nil && err != http.ErrServerClosed {
errCh <- err
}
close(errCh)
}()
// Block until signal or server error
select {
case <-ctx.Done():
logger.Info("shutdown signal received")
// Fail readiness immediately
ready = nil
case err := <-errCh:
return err
}
// Drain with timeout independent of parent ctx
shutdownCtx, cancel := context.WithTimeout(
context.Background(), 25*time.Second,
)
defer cancel()
if err := srv.Shutdown(shutdownCtx); err != nil {
return fmt.Errorf("server shutdown: %w", err)
}
logger.Info("server stopped gracefully")
return nil
}Closing downstream resources safely
After srv.Shutdown() returns, all HTTP handlers have completed. Only then should you close database pools, Redis connections, and Kafka consumers. Closing these before shutdown completes causes panics in handlers still executing queries. Structure your cleanup as deferred functions registered after server creation, or use an explicit resource manager that respects ordering.
Why does my Go service still drop requests during deployment?
If you have implemented everything above and still see 502/504 errors, the issue usually lies outside your Go code. Check these common failure points:
- terminationGracePeriodSeconds too short: If your longest request takes 20s but grace period is 15s, Kubernetes sends SIGKILL. Measure p99 latency and add buffer.
- Missing preStop sleep: Endpoint propagation lag means new requests arrive after SIGTERM. The 3–5s sleep absorbs this.
- Readiness not failing atomically: Using a mutex instead of atomic/channel introduces a window where readiness returns 200 after signal receipt.
- Load balancer health check interval too long: Cloud LBs may cache healthy status for 30s+. Align cloud LB health check frequency with Kubernetes probe timing.
- Connection reuse without notification: HTTP/2 persistent connections may not respect GOAWAY frames properly in older clients. Consider disabling keep-alive during shutdown for sensitive workloads.
Debugging these issues requires correlating application logs with infrastructure events. Structured logging with request IDs, as covered in structured logging best practices, makes it possible to trace exactly which requests were in-flight during termination and whether they completed or were severed.
Implementing Graceful Shutdown and Health Checks in Go Reliably
Production-grade graceful shutdown and health checks in Go demand treating application code and Kubernetes configuration as a single unit. Implement signal-aware contexts, separate liveness from readiness with correct shutdown semantics, configure probe timing to match your workload characteristics, and always close downstream resources after HTTP draining completes. Test this end-to-end in a staging cluster with realistic traffic before relying on it in production. If your team needs help auditing your current deployment pipeline or designing compliant infrastructure for regulated workloads, reach out to discuss your specific architecture.
