Graceful Shutdown and Health Checks in Elixir

By Khimananda Oli | Last reviewed: August 2026

Dropped requests during rolling deployments remain one of the most common reliability failures I see in Elixir production environments. While the BEAM VM is famous for concurrency, achieving true zero-downtime requires explicit configuration for graceful shutdown and health checks in Elixir. Without proper signal handling and readiness probes, load balancers will route traffic to terminating pods or unready instances, causing intermittent 502 errors that are notoriously difficult to debug. This guide covers the exact OTP, Phoenix, and Kubernetes configurations needed to make your Elixir applications deploy safely.

How does graceful shutdown work in Elixir OTP applications?

The BEAM virtual machine handles operating system signals differently than typical runtime environments. When Kubernetes or systemd sends a SIGTERM, the Erlang runtime translates this into an internal shutdown sequence rather than immediate termination. Understanding this mechanism is critical because misconfiguration here is the root cause of most deploy-time errors.

Your application's top-level supervisor receives the shutdown signal first. It then initiates a reverse-order shutdown of its children based on the supervision tree structure. Each child process gets a chance to handle the :shutdown message in its terminate/2 callback. The key constraint is the shutdown timeout value defined in each child specification.

Configuring shutdown timeouts in child specs

The default shutdown timeout in Elixir is 5,000 milliseconds. For web applications handling database transactions or external API calls, this is almost always insufficient. You must explicitly set higher values:

# lib/my_app/application.ex
children = [
  {MyApp.Repo, []},
  {MyAppWeb.Endpoint, [shutdown: 30_000]},
  {MyApp.Worker, [shutdown: 60_000]}
]

opts = [strategy: :one_for_one, name: MyApp.Supervisor]
Supervisor.start_link(children, opts)

A common mistake is setting the endpoint shutdown but forgetting long-running GenServers. If a worker has a 60-second cleanup requirement but uses the default 5-second timeout, the supervisor will forcefully kill it mid-operation. Always audit every child spec for realistic drain times. For deeper context on how these processes interact with databases, review PostgreSQL administration essentials to understand connection pool behavior during termination.

Handling terminate callbacks correctly

Processes must trap exits to execute their terminate/2 callback. Without this flag, the process dies immediately upon receiving the shutdown signal:

defmodule MyApp.Worker do
  use GenServer

  def init(state) do
    Process.flag(:trap_exit, true)
    {:ok, state}
  end

  def terminate(:shutdown, state) do
    # Complete pending work, flush buffers, close connections
    MyApp.Buffer.flush(state.buffer)
    :ok
  end
end

Note that terminate/2 is not guaranteed to run if the node crashes or receives SIGKILL. Never rely on it for critical data persistence; use it only for best-effort cleanup and connection draining.

How do you configure Phoenix endpoints for zero-downtime deploys?

Phoenix requires specific configuration to stop accepting new connections while allowing in-flight requests to complete. This two-phase approach is what actually enables zero-downtime behavior at the HTTP layer.

Endpoint shutdown configuration

In your production config, set the endpoint shutdown timeout to match your longest expected request duration plus a safety margin:

# config/prod.exs
config :my_app, MyAppWeb.Endpoint,
  server: true,
  shutdown: 30_000,  # 30 seconds to drain active requests
  drainer: [{MyAppWeb.Endpoint, 5_000}]  # Optional: staged drain

The drainer option allows phased shutdown where the endpoint stops accepting on port 4000 first, waits for existing connections to settle, then proceeds with full shutdown. This is particularly valuable when sitting behind a load balancer with slow connection recycling.

Connection draining with Plug.Cowboy

If using Plug.Cowboy (the default), understand that it maintains a list of active connections. During shutdown, it stops the listener socket immediately but keeps worker processes alive until the timeout expires. Requests already being processed continue normally; new connections are refused at the TCP level.

For applications using Bandit instead of Cowboy, the shutdown semantics are similar but the configuration keys differ slightly. Always verify against your specific adapter's documentation. The critical point is that the HTTP server must be told explicitly to drain rather than terminate abruptly.

What makes an effective health check endpoint in Elixir?

Health checks serve two distinct purposes that many teams conflate: liveness (is the process running?) and readiness (can it serve traffic?). Your Elixir application needs separate endpoints for each, and confusing them causes cascading failures during deploys.

Implementing a lightweight liveness probe

The liveness endpoint should return 200 as fast as possible with zero external dependencies. Its sole purpose is confirming the BEAM scheduler is responsive:

# lib/my_app_web/controllers/health_controller.ex
defmodule MyAppWeb.HealthController do
  use MyAppWeb, :controller

  def live(conn, _params) do
    conn
    |> put_status(200)
    |> json(%{status: "ok"})
  end
end

Do not query the database, check Redis, or call any external service here. If your database goes down temporarily, a heavy liveness check will fail, causing Kubernetes to restart your pod unnecessarily. This creates a restart storm that makes the outage worse. Liveness means "is the Erlang VM running," nothing more.

Building a comprehensive readiness probe

Readiness determines whether the pod should receive traffic. This is where you verify actual operational capacity:

def ready(conn, _params) do
  checks = %{
    database: check_database(),
    redis: check_redis(),
    vault: check_vault_secrets()
  }

  status = if Enum.all?(checks, fn {_k, v} -> v == :ok end) do
    200
  else
    503
  end

  conn
  |> put_status(status)
  |> json(%{status: status_text(status), checks: checks})
end

defp check_database do
  case Ecto.Adapters.SQL.query(MyApp.Repo, "SELECT 1", []) do
    {:ok, _} -> :ok
    {:error, _reason} -> :error
  end
rescue
  _ -> :error
end

Set a strict timeout on each check (2-3 seconds). A readiness probe that hangs blocks the entire deployment pipeline. For teams managing complex data layers, understanding MongoDB administration basics helps design appropriate connectivity checks that don't themselves become bottlenecks.

How do you align Kubernetes termination settings with Elixir shutdown?

Your Elixir configuration is meaningless if Kubernetes kills the container before the application finishes draining. The three critical values must satisfy: preStop + shutdown_timeout <= terminationGracePeriodSeconds.

Kubernetes manifest alignment

spec:
  terminationGracePeriodSeconds: 60
  containers:
  - name: elixir-app
    lifecycle:
      preStop:
        exec:
          command: ["/bin/sh", "-c", "sleep 5"]
    readinessProbe:
      httpGet:
        path: /health/ready
        port: 4000
      initialDelaySeconds: 10
      periodSeconds: 5
      timeoutSeconds: 3
      failureThreshold: 3
    livenessProbe:
      httpGet:
        path: /health/live
        port: 4000
      initialDelaySeconds: 15
      periodSeconds: 10
      timeoutSeconds: 3

The preStop sleep is essential. When Kubernetes marks a pod as terminating, it simultaneously removes the endpoint from the service and sends SIGTERM. Due to iptables/ipvs propagation delay, some requests may still arrive after SIGTERM. The 5-second sleep gives the network stack time to converge before your application stops accepting. Skipping this step causes the exact 502 errors you're trying to prevent.

Common timing mismatches

I've seen teams set terminationGracePeriodSeconds to 300 "to be safe," then wonder why scale-down events take five minutes. Right-size this value based on actual p99 request latency plus drain overhead. Monitoring these timings properly requires solid observability; consider reviewing the four golden signals of monitoring to establish baselines for your drain duration metrics.

Why do Elixir applications still drop requests despite correct configuration?

Even with perfect configuration, several subtle issues cause request loss. These are the failure modes I encounter most frequently in production audits.

• Connection pool exhaustion during drain: As workers shut down, remaining active requests compete for fewer available connections. If your pool size equals your concurrent request limit exactly, late requests will timeout waiting for a connection that will never free up. Set pool size 20-30% above expected peak concurrency.
• Async tasks not linked to supervisor: Tasks spawned via Task.async/1 without proper supervision won't receive shutdown signals. They continue running until the OS kills the entire container. Always use Task.Supervisor.async_nolink/3 for fire-and-forget work that should respect shutdown.
• NIF/native code blocking the scheduler: If you use NIFs that block for extended periods, the BEAM cannot process the shutdown signal promptly. The shutdown timer starts when the signal is received, not when the scheduler becomes free to handle it. Profile any native extensions for scheduler blocking.
• Load balancer health check interval too long: If your cloud load balancer checks health every 30 seconds but your pod terminates in 10, there's a 20-second window where dead pods still receive traffic. Align LB health intervals with your deployment velocity.

Validating your shutdown behavior

Don't assume your configuration works—test it. Use a tool like k6 or vegeta to send continuous traffic while triggering pod termination:

# Terminal 1: Continuous load
k6 run --duration 5m load-test.js

# Terminal 2: Trigger shutdown
kubectl delete pod my-app-xyz --grace-period=60

# Observe: Zero 5xx responses should appear in k6 output

Add structured logging to your terminate/2 callbacks to track which processes completed cleanup versus timed out. This telemetry is invaluable during incident investigation. If you're building out broader observability, the patterns in structured logging best practices apply directly to shutdown instrumentation.

Deploy Reliable Elixir Applications With Confidence

Getting graceful shutdown and health checks in Elixir right requires coordination across three layers: OTP supervision configuration, Phoenix endpoint tuning, and Kubernetes manifest alignment. Missing any one layer breaks the chain. Start by auditing your current shutdown timeouts against actual p99 latencies, implement separate liveness and readiness endpoints, and validate with controlled chaos testing in staging before touching production. If your team needs help designing deployment-safe Elixir architectures or auditing existing systems for shutdown correctness, reach out to discuss your infrastructure.