Run AI/ML Workloads on Kubernetes with GPUs

By Khimananda Oli | Last reviewed: August 2026

To successfully run AI/ML workloads on Kubernetes with GPUs, you must bridge the gap between container orchestration and bare-metal hardware acceleration. Standard Kubernetes schedulers do not understand GPU topology or VRAM capacity natively; they rely entirely on vendor-specific device plugins to expose accelerator resources as schedulable units. This guide provides the exact configuration patterns, driver prerequisites, and operational guardrails needed to deploy stable machine learning inference and training jobs in production environments.

How Do You Configure Kubernetes Nodes to Run AI/ML Workloads with GPUs?

Before a single pod can request an accelerator, the underlying node must be correctly provisioned. A common mistake I see in teams attempting to understand GPU requirements for AI is assuming that installing the NVIDIA drivers on the host OS is sufficient. In Kubernetes, the container runtime (containerd or CRI-O) requires specific integration to pass through PCI devices safely.

Install the NVIDIA GPU Operator

In 2026, the recommended approach for most clusters is the NVIDIA GPU Operator. It automates the installation of drivers, the container toolkit, and the device plugin via a single Helm chart. This ensures version alignment between the kernel module and the userspace libraries, which is critical for avoiding silent failures during model loading.

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set devicePlugin.enabled=true

If you are running on managed services like Amazon EKS or Azure AKS, verify whether the cloud provider's optimized AMI already includes the necessary components. On EKS, for example, you often only need the device plugin, as the AMI contains the driver and runtime. Installing the full operator on such nodes can cause conflicts. Always check your specific EKS deployment guide or cloud documentation before applying manifests.

Verify Node Readiness

After installation, validate that the node reports allocatable GPU resources. The following command should return a non-zero integer for each GPU-equipped node:

kubectl get nodes -o json | jq '.items[] | select(.status.allocatable."nvidia.com/gpu" != null) | {name: .metadata.name, gpus: .status.allocatable."nvidia.com/gpu"}'

If this returns empty, the device plugin daemonset is likely failing due to missing kernel headers or incompatible CUDA versions. Check the gpu-operator-validator pods for detailed diagnostic logs.

What Is the Correct Pod Spec for Requesting GPU Resources?

Kubernetes treats GPUs as an "extended resource." Unlike CPU and memory, you cannot specify fractional GPUs (e.g., 0.5 nvidia.com/gpu) unless you have explicitly configured MIG (Multi-Instance GPU) partitioning. For standard deployments, requests and limits must match exactly, and the value must be an integer.

Basic Single-GPU Inference Deployment

When defining your workload, always set both requests and limits to the same value. The scheduler uses the request to place the pod, but the limit enforces isolation. Omitting the limit can lead to unpredictable behavior where a pod accesses unassigned devices.

apiVersion: apps/v1
kind: Deployment
metadata:
  name: llm-inference-service
spec:
  replicas: 2
  selector:
    matchLabels:
      app: llm-inference
  template:
    metadata:
      labels:
        app: llm-inference
    spec:
      nodeSelector:
        nvidia.com/gpu.product: "NVIDIA-A10G"
      containers:
      - name: vllm-worker
        image: vllm/vllm-openai:v0.6.0
        args: ["--model", "meta-llama/Llama-3-8B-Instruct"]
        resources:
          limits:
            nvidia.com/gpu: 1
            memory: 24Gi
          requests:
            nvidia.com/gpu: 1
            memory: 24Gi
        env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: "all"
        ports:
        - containerPort: 8000

Handling Multi-GPU Training Jobs

For distributed training or large language models requiring tensor parallelism, request multiple GPUs in a single pod. Ensure you also configure the appropriate environment variables for NCCL communication. Without NVIDIA_VISIBLE_DEVICES=all, some frameworks fail to initialize inter-GPU links even when multiple devices are allocated.

• Tensor Parallelism: Set nvidia.com/gpu: 4 for models like Llama-3-70B that require sharding across four A10Gs.
• Data Parallelism: Prefer separate single-GPU pods with a distributed training operator (like Kubeflow Training Operator) over multi-GPU pods for better fault isolation.
• Shared Memory: Add an emptyDir volume mounted at /dev/shm with sufficient size. PyTorch DataLoader and NCCL require shared memory for efficient inter-process communication; the default 64MB is insufficient for ML workloads.

How Do You Monitor GPU Utilization for ML Workloads in Production?

Scheduling the GPU is only half the battle; ensuring it is actually being used efficiently is where most teams struggle. Standard Kubernetes metrics do not include GPU telemetry. You must deploy the NVIDIA DCGM Exporter to expose Prometheus-compatible metrics. This is essential if you want to implement effective metric-based monitoring for expensive accelerator infrastructure.

Key Metrics to Track

Do not just monitor utilization percentage. High utilization does not always mean efficiency—it could indicate a memory bottleneck causing thrashing. Focus on these three signals:

Integrating with Prometheus Stack

The DCGM exporter runs as a DaemonSet alongside the device plugin. Configure your Prometheus ServiceMonitor to scrape port 9400. Once ingested, build Grafana dashboards that overlay GPU metrics with application latency. If you are using OpenTelemetry for application tracing, correlate high-latency spans with GPU memory pressure events to identify bottlenecks in your inference pipeline. Refer to our OpenTelemetry implementation guide for linking infrastructure metrics to application traces.

How Do You Optimize Costs When Running GPU Workloads on Kubernetes?

GPUs are typically the most expensive line item in any ML infrastructure budget. Leaving them idle during development cycles or due to poor scheduling is financially unsustainable. Cost optimization requires a combination of technical configuration and operational policy.

Implement Cluster Autoscaling with GPU Awareness

Standard cluster autoscalers may not scale down GPU nodes aggressively enough because they treat them like regular compute. Configure the Cluster Autoscaler with explicit GPU scaling policies. On AWS EKS, use Karpenter with NodePools that define GPU instance types separately from general-purpose nodes. This allows rapid provisioning of spot instances for training jobs while maintaining on-demand base capacity for inference.

Use Time-Sharing for Development Environments

For non-production workloads like Jupyter notebooks or experimentation, strict 1:1 GPU allocation is wasteful. Enable GPU Time-Slicing in the device plugin configuration. This allows multiple pods to share a single physical GPU by interleaving execution. While this introduces context-switching overhead unsuitable for production inference, it can reduce dev/test costs by 4x–8x.

# Example ConfigMap for time-slicing (dev clusters only)
apiVersion: v1
kind: ConfigMap
metadata:
  name: device-plugin-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
        - name: nvidia.com/gpu
          replicas: 4  # Each physical GPU appears as 4 virtual devices

Right-Size Your Persistent Storage

ML workloads are I/O intensive. Using network-attached storage (EBS/GP3) for model checkpoints and dataset caching creates latency that keeps GPUs idle waiting for data. For high-performance training, use local NVMe SSDs via the local persistent volume provisioner. While less durable, the throughput difference directly translates to GPU utilization efficiency. Always pair local storage with automated checkpointing to object storage (S3/GCS) to prevent data loss on node termination.

Operational Checklist for Production GPU Clusters

Successfully operating GPU clusters requires discipline beyond initial setup. Use this checklist to validate your environment before onboarding critical workloads:

1. Driver/Runtime Alignment: Verify CUDA driver versions match across all nodes in a pool. Mixed versions cause intermittent pod failures that are difficult to debug.
2. Resource Quotas: Implement namespace-level quotas for nvidia.com/gpu to prevent runaway training jobs from starving inference services.
3. Image Optimization: Use multi-stage builds to keep ML images under 5GB where possible. Large images increase cold-start times significantly on GPU nodes due to slower pull rates on specialized instance types.
4. Health Checks: Configure liveness probes that verify GPU accessibility, not just HTTP response. A pod can return 200 OK while its GPU context has crashed silently.
5. Drain Safety: Ensure Pod Disruption Budgets account for long-running training jobs. Use checkpointing mechanisms so interrupted jobs can resume without losing hours of compute.

Next Steps for Your GPU Infrastructure

Running AI/ML workloads on Kubernetes with GPUs transforms your cluster from a generic orchestration platform into a specialized AI factory. The key is treating GPU resources with the same rigor as database connections or API keys: explicit allocation, continuous monitoring, and strict lifecycle management. Start by validating your device plugin installation and establishing baseline utilization metrics before scaling out. If your team needs help designing a compliant, cost-efficient GPU infrastructure or auditing existing ML pipelines for production readiness, reach out to discuss your architecture.