Serve ML Models with GPU on Kubernetes

By Khimananda Oli | Last reviewed: August 2026

Deploying machine learning workloads requires more than just containerization; you need hardware-aware orchestration to avoid burning budget on idle accelerators. When you serve ML models with GPU on Kubernetes, the cluster must correctly expose hardware resources, manage proprietary drivers, and scale pods based on inference queue depth rather than CPU utilization. This guide covers the production-grade architecture required to run LLMs and computer vision models reliably, drawing from patterns I use daily across AWS EKS and on-premise clusters.

How do you configure Kubernetes nodes to expose GPU resources?

Kubernetes does not detect GPUs natively. Without additional configuration, your pods will fail to schedule with Insufficient nvidia.com/gpu errors even if the underlying EC2 or bare-metal instance has four A10Gs installed. The industry standard in 2026 is the NVIDIA GPU Operator, which automates the entire stack lifecycle including drivers, container runtime, device plugin, and monitoring exporters.

Installing the NVIDIA GPU Operator via Helm

Avoid manual driver installation on nodes. Manual installs break during kernel upgrades and create drift between instances. The operator runs as a DaemonSet and handles node labeling, driver compatibility checks, and MIG (Multi-Instance GPU) partitioning automatically.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install --wait --generate-name \
  -n gpu-operator --create-namespace \
  nvidia/gpu-operator \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set dcgmExporter.enabled=true

After installation, verify that nodes are advertising GPU capacity. This confirmation step is critical before deploying any inference workload.

kubectl get nodes -o custom-columns=\
NAME:.metadata.name,\
GPU:.status.allocatable.nvidia\.com/gpu,\
READY:.status.conditions[-1].type

# Expected output:
# NAME                     GPU   READY
# ip-10-0-1-45.ec2         4     Ready
# ip-10-0-1-78.ec2         4     Ready

If you operate in Nepal or regions with restricted internet connectivity, pre-pull all operator images into a private registry like ECR or Harbor before installation. The operator attempts to pull ~2GB of driver containers on first launch, which can timeout on slower links. Configure driver.repository and toolkit.repository values in your Helm release to point to your local mirror. For teams managing multiple clusters, see my guide on managing multiple Kubernetes clusters with Rancher to apply GPU operator configurations consistently across environments.

Which inference server should you deploy for LLMs and vision models?

Serving frameworks matter more than raw GPU specs. A naive Flask wrapper around PyTorch will leave 70% of your accelerator idle while waiting for Python GIL locks. In production, you need continuous batching, PagedAttention memory management, and speculative decoding support.

Deploying vLLM with Correct Resource Boundaries

Always set both requests and limits to identical values for GPU resources. Unlike CPU, GPUs cannot be oversubscribed safely in most inference scenarios. Fractional GPUs require MIG or time-slicing configuration in the operator.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llama3-8b-instruct
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llama3-8b
  template:
    metadata:
      labels:
        app: llama3-8b
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.8.5
        args:
          - "--model"
          - "meta-llama/Meta-Llama-3-8B-Instruct"
          - "--tensor-parallel-size"
          - "1"
          - "--max-model-len"
          - "4096"
          - "--gpu-memory-utilization"
          - "0.90"
        ports:
        - containerPort: 8000
        resources:
          requests:
            nvidia.com/gpu: 1
            memory: "24Gi"
            cpu: "4"
          limits:
            nvidia.com/gpu: 1
            memory: "24Gi"
            cpu: "4"
        env:
        - name: HUGGING_FACE_HUB_TOKEN
          valueFrom:
            secretKeyRef:
              name: hf-token
              key: token
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10

Note the --gpu-memory-utilization flag set to 0.90. Never set this to 1.0. Leave headroom for CUDA context allocation and unexpected KV cache spikes during long-context requests. Setting it too high causes silent OOM kills that manifest as intermittent 500 errors under load, not immediate crashes. Understanding these resource boundaries ties directly into Kubernetes resource limits and requests best practices.

How do you autoscale GPU workloads based on inference demand?

Standard Horizontal Pod Autoscaler (HPA) uses CPU or memory metrics. These are useless for GPU inference. A vLLM pod can sit at 5% CPU utilization while its GPU is completely saturated processing a batch of 32 concurrent requests. Conversely, CPU might spike during tokenization while the GPU sits idle. You need queue-aware scaling.

Configuring KEDA for Queue-Depth Scaling

KEDA (Kubernetes Event-Driven Autoscaling) polls Prometheus metrics exposed by your inference server and adjusts replica counts accordingly. vLLM exposes vllm:num_requests_waiting and vllm:gpu_cache_usage_perc natively on the /metrics endpoint.

1. Install KEDA and the Prometheus adapter in your cluster.
2. Create a ScaledObject targeting your inference deployment.
3. Set scale-up thresholds based on acceptable latency SLOs, not arbitrary percentages.
4. Configure cooldown periods to prevent oscillation during traffic bursts.

apiVersion: keda.sh/v1alpha1
kind: ScaledObject
metadata:
  name: llama3-scaler
spec:
  scaleTargetRef:
    name: llama3-8b-instruct
  minReplicaCount: 1
  maxReplicaCount: 8
  pollingInterval: 10
  cooldownPeriod: 300
  triggers:
  - type: prometheus
    metadata:
      serverAddress: http://prometheus.monitoring:9090
      metricName: gpu_queue_depth
      query: avg(vllm:num_requests_waiting{app="llama3-8b"})
      threshold: "4"
      activationThreshold: "1"

The cooldownPeriod: 300 is deliberate. GPU pods take 45–90 seconds to start and load weights. Aggressive scale-down causes thrashing where pods terminate before finishing warm-up, then immediately rescale. Five minutes of cool-down absorbs transient dips without sacrificing responsiveness to sustained load. For deeper coverage on defining meaningful thresholds, read defining meaningful SLIs and SLOs for ML services.

What are common pitfalls when running GPU workloads in production?

I have debugged dozens of GPU deployments across AWS, Azure, and on-prem data centers. The same three issues appear repeatedly regardless of cloud provider or model size.

• Cold start latency dominates user experience. Loading a 70B parameter model takes 3–5 minutes from S3. Pre-warm pods using init containers that fetch weights to a shared PersistentVolume backed by NVMe storage. Use Longhorn distributed storage or AWS FSx for Lustre to achieve sub-second weight mounting across nodes.
• Driver version mismatches after node rotation. When cluster autoscaler adds new GPU nodes, they may pull different driver versions than existing ones. Pin driver versions explicitly in the GPU Operator Helm values (driver.version: "550.54.15") rather than relying on latest tags.
• Ignoring DCGM exporter metrics. Without DCGM, you are flying blind. GPU utilization, memory bandwidth, SM occupancy, and power draw are only visible through DCGM. Install the exporter as part of the operator and build Grafana dashboards before your first production incident. Reference the Prometheus and Grafana monitoring stack guide for dashboard templates adapted for GPU metrics.

Security also demands attention. GPU workloads often process sensitive data. Apply Kubernetes network policies to restrict egress from inference pods. Models should only communicate with the model registry and observability backends. Block all other outbound traffic to prevent data exfiltration through compromised inference endpoints.

Production Checklist for GPU-Accelerated Inference

Before promoting any GPU workload to production, verify these operational requirements. Skipping any single item has caused incidents in environments I have audited.

• Persistent model caching: Weights stored on ephemeral disk cause 3+ minute cold starts on every pod restart. Use shared PVs with ReadWriteMany access.
• Explicit tolerations: GPU nodes often carry taints (nvidia.com/gpu=present:NoSchedule). Ensure deployments include matching tolerations or scheduling fails silently.
• Graceful shutdown hooks: Set terminationGracePeriodSeconds: 120. In-flight generations need time to complete. Default 30s kills active requests mid-stream.
• Cost allocation tags: Tag GPU node groups with project/team identifiers. Untagged GPU spend becomes untraceable within weeks.
• Disaster recovery: GPU instances have limited availability zones. Define fallback regions or instance types in your Cluster Autoscaler configuration.

Next Steps for Your GPU Infrastructure

When you serve ML models with GPU on Kubernetes successfully, the infrastructure fades into the background and teams focus on model quality instead of firefighting scheduling errors. Start with the NVIDIA GPU Operator and vLLM as your baseline, instrument everything with DCGM metrics, and implement KEDA scaling before your first traffic spike hits. If your team needs help designing audit-ready GPU infrastructure or optimizing inference costs across multi-cloud environments, reach out to discuss your specific architecture.