Scale and Monitor Phoenix in Production

By Khimananda Oli | Last reviewed: August 2026

You have shipped your Elixir application, but traffic growth exposes latency spikes and opaque failures that local testing never revealed. To successfully scale and monitor Phoenix in production, you must treat the BEAM VM as a distributed system requiring explicit resource contracts, structured telemetry, and infrastructure-aware autoscaling. This guide moves beyond basic configuration to the operational patterns required for high-traffic systems, integrating insights from the four golden signals of monitoring to keep your service reliable under load.

How do you tune the BEAM VM for containerized Phoenix deployments?

The most common failure mode when deploying Elixir to Docker or Kubernetes is mismatched scheduler counts. By default, the BEAM creates one scheduler per logical core detected on the host. In a containerized environment, this often means detecting 64 cores on the underlying node while your pod has a 2-core limit. The result is massive context-switching overhead and unpredictable latency.

Aligning Schedulers with Resource Limits

You must explicitly tell the BEAM how many schedulers to create based on your container's CPU limit, not the host's capacity. For a pod with a 2-core limit, set the schedulers to 2. This ensures each scheduler maps cleanly to an available vCPU without fighting the CFS bandwidth controller.

# config/runtime.exs
if System.get_env("PHX_SERVER") do
  # Match schedulers to container CPU limit
  # Use +S for total schedulers, +SDcpu for dirty CPU schedulers
  erl_opts = "+S 2:2 +SDcpu 2:2 +Q 65536"
  
  config :my_app, MyAppWeb.Endpoint,
    http: [
      port: String.to_integer(System.get_env("PORT") || "4000"),
      transport_options: [
        max_connections: 16384,
        num_acceptors: 100
      ]
    ]
end

Beyond schedulers, increase the async thread pool size if your application performs blocking NIF calls or file I/O. The default async threads are insufficient for high-throughput production workloads. Setting +A 16 or higher prevents these operations from blocking the main schedulers. Always validate these settings using :erlang.system_info(:schedulers) in a remote console session after deployment to confirm they took effect.

Memory Allocation Strategy

Container memory limits require careful allocator tuning. The BEAM's default allocators can fragment memory in ways that trigger OOM kills even when heap usage appears low. Configure the multi-block allocator to be more conservative with virtual memory reservations:

• Use +MMmcs 30 to cap the number of memory carriers
• Set +MHlmbcs 512 to control large block carrier sizes
• Enable +MMscs 10 for super carrier reservation alignment

These flags prevent the BEAM from reserving excessive virtual address space that exceeds your container's cgroup memory limit. Test thoroughly under load, as improper allocator settings can degrade performance or cause crashes during garbage collection cycles.

What metrics should you expose to monitor Phoenix health?

Monitoring Phoenix requires exposing both BEAM internals and application-specific business metrics. Relying solely on HTTP status codes misses queue saturation, message mailbox growth, and connection pool exhaustion—the actual precursors to outages. Following principles from Prometheus metrics fundamentals, structure your telemetry around actionable signals.

Essential BEAM Metrics

Your monitoring stack must ingest these core VM metrics at minimum:

Application-Level Telemetry

Attach telemetry handlers in your supervision tree to capture domain-specific events. Phoenix provides built-in telemetry for endpoint requests, LiveView lifecycle, and Ecto queries. Extend this with custom events for critical business operations:

# lib/my_app/telemetry.ex
defmodule MyApp.Telemetry do
  use Supervisor
  import Telemetry.Metrics

  def start_link(arg) do
    Supervisor.start_link(__MODULE__, arg, name: __MODULE__)
  end

  @impl true
  def init(_arg) do
    children = [
      {:telemetry_poller, measurements: periodic_measurements(), period: 10_000},
      {TelemetryMetricsPrometheus, [metrics: metrics()]}
    ]

    Supervisor.init(children, strategy: :one_for_one)
  end

  defp metrics do
    [
      # Phoenix request duration histogram
      distribution("phoenix.endpoint.stop.duration",
        tags: [:method, :route],
        buckets: [10, 50, 100, 250, 500, 1000, 2500]
      ),
      
      # Custom business metric: payment processing time
      distribution("my_app.payment.process.duration",
        tags: [:provider, :status],
        unit: {:native, :millisecond}
      ),
      
      # Connection pool utilization
      last_value("db.repo.pool.size", tags: [:repo]),
      last_value("db.repo.pool.checked_out", tags: [:repo])
    ]
  end
end

Always use histograms or distributions for latency metrics, never averages. Averages hide tail latency that destroys user experience. Tag metrics with cardinality-safe labels; avoid unbounded tags like user IDs or request UUIDs that will explode your Prometheus storage.

How do you implement distributed tracing in Phoenix with OpenTelemetry?

Metrics tell you something is wrong; traces show you where. Modern Phoenix applications integrate with OpenTelemetry to propagate context across services, databases, and external APIs. This is essential for debugging latency in microservice architectures or complex LiveView interactions. Refer to instrumenting apps with OpenTelemetry for foundational concepts.

Setting Up OpenTelemetry in Phoenix

Add the required dependencies and configure automatic instrumentation:

# mix.exs
defp deps do
  [
    {:opentelemetry, "~> 1.3"},
    {:opentelemetry_exporter, "~> 1.6"},
    {:opentelemetry_phoenix, "~> 1.1"},
    {:opentelemetry_ecto, "~> 1.1"},
    {:opentelemetry_reqwest, "~> 0.2"}
  ]
end

In your application supervisor, attach the instrumentation modules before your endpoint starts. This ensures spans are created for every incoming request and database query automatically:

# lib/my_app/application.ex
def start(_type, _args) do
  OpentelemetryPhoenix.setup(adapter: :bandit)
  OpentelemetryEcto.setup([:my_app, :repo])
  
  children = [
    MyApp.Repo,
    MyAppWeb.Endpoint
  ]
  
  opts = [strategy: :one_for_one, name: MyApp.Supervisor]
  Supervisor.start_link(children, opts)
end

Adding Custom Spans for Business Logic

Automatic instrumentation covers framework boundaries, but your critical business logic needs manual spans. Wrap expensive operations to correlate them with request traces:

defmodule MyApp.Billing do
  require OpenTelemetry.Tracer, as: Tracer
  
  def process_payment(order) do
    Tracer.with_span "billing.process_payment" do
      Tracer.set_attribute("order.id", order.id)
      Tracer.set_attribute("payment.provider", "stripe")
      
      # Your payment logic here
      result = Stripe.Charge.create(...)
      
      case result do
        {:ok, charge} ->
          Tracer.set_attribute("payment.success", true)
          {:ok, charge}
        {:error, reason} ->
          Tracer.set_status(:error, inspect(reason))
          {:error, reason}
      end
    end
  end
end

Export traces to Tempo, Jaeger, or Datadog depending on your existing observability stack. Ensure your sampling rate is appropriate for production; head-based sampling at 10% is typical for high-traffic services to manage cost while retaining sufficient signal for debugging.

When should you use Kubernetes HPA versus vertical scaling for Phoenix?

Choosing between horizontal and vertical scaling depends on your workload characteristics. Phoenix applications benefit uniquely from horizontal scaling due to the BEAM's lightweight process model, but vertical scaling has its place for stateful components or single-node deployments common in Nepal's cost-sensitive hosting environments.

Decision Framework

Configuring Custom Metric HPA

Never scale Phoenix pods based on CPU alone. The BEAM saturates all available schedulers during normal operation, making CPU a poor proxy for actual overload. Instead, scale on request rate or queue depth using the Prometheus adapter:

# k8s/hpa.yaml
apiVersion: autoscaling/v2
kind: HorizontalPodAutoscaler
metadata:
  name: phoenix-app-hpa
spec:
  scaleTargetRef:
    apiVersion: apps/v1
    kind: Deployment
    name: phoenix-app
  minReplicas: 2
  maxReplicas: 20
  metrics:
  - type: Pods
    pods:
      metric:
        name: http_requests_per_second
      target:
        type: AverageValue
        averageValue: "1000"
  behavior:
    scaleUp:
      stabilizationWindowSeconds: 60
      policies:
      - type: Pods
        value: 2
        periodSeconds: 60
    scaleDown:
      stabilizationWindowSeconds: 300
      policies:
      - type: Pods
        value: 1
        periodSeconds: 120

The stabilization windows prevent flapping during transient traffic bursts. Set scale-down windows longer than scale-up to avoid premature termination during gradual traffic decline. Always maintain at least two replicas for zero-downtime deployments and rolling updates.

How do you handle graceful shutdowns and connection draining?

Phoenix applications serving long-lived WebSocket connections or SSE streams require careful shutdown handling to avoid dropping active clients during deployments or scale-down events. The BEAM's default shutdown timeout is often too aggressive for production workloads.

Configuring Endpoint Shutdown

Extend the shutdown window in your endpoint configuration to allow in-flight requests and WebSocket connections to complete:

# config/runtime.exs
config :my_app, MyAppWeb.Endpoint,
  server: true,
  # Allow 30 seconds for graceful drain
  shutdown_timeout: 30_000,
  # Signal readiness to k8s only after warmup
  check_origin: false

# In your endpoint module
defmodule MyAppWeb.Endpoint do
  use Phoenix.Endpoint, otp_app: :my_app
  
  # Pre-warm caches before accepting traffic
  def init(:supervisor, config) do
    MyApp.Cache.warm()
    {:ok, config}
  end
end

Pair this with Kubernetes preStop hooks and readiness probes. The preStop hook should sleep for 5-10 seconds to allow the load balancer to remove the pod from rotation before SIGTERM arrives. This prevents the race condition where new requests arrive during the brief window between pod termination signal and LB update propagation.

LiveView and WebSocket Draining

For LiveView applications, implement connection draining by tracking active sockets and delaying shutdown until they disconnect or migrate. Use Phoenix.PubSub to broadcast shutdown warnings to connected clients, allowing them to reconnect to other nodes gracefully. This pattern is critical for real-time dashboards and collaborative features where abrupt disconnections destroy user trust.

Operationalizing Phoenix Reliability

To reliably scale and monitor Phoenix in production, treat observability and capacity planning as first-class engineering concerns, not afterthoughts. Start by aligning your BEAM scheduler configuration with container limits, then build comprehensive telemetry covering both VM internals and business-critical paths. Implement OpenTelemetry tracing early—retrofitting it later is painful. Choose your scaling strategy based on state requirements and traffic patterns, always validating with load tests that mirror production characteristics. If your team needs help designing audit-ready, observable Phoenix infrastructure that passes compliance reviews and handles real-world traffic, reach out to discuss your architecture.