GPU Scheduling on Kubernetes

By Khimananda Oli | Last reviewed: August 2026

Running AI or ML workloads without proper GPU scheduling on Kubernetes leads to stranded hardware, pending pods, and unpredictable training times. GPUs are expensive, stateful resources that the default kube-scheduler cannot see without specific extensions and configuration. This guide covers the exact device plugin architecture, resource declaration patterns, and sharing mechanisms you need to run production-grade accelerated workloads reliably.

How does GPU scheduling on Kubernetes actually work?

Kubernetes does not natively understand GPUs. The core scheduler only knows about CPU, memory, and ephemeral storage. To enable GPU scheduling on Kubernetes, you must deploy a Device Plugin that implements the DevicePlugin gRPC interface. This plugin runs as a DaemonSet on every GPU-equipped node and performs three critical functions: discovery, registration, and allocation.

During discovery, the plugin scans the PCIe bus or uses vendor libraries (like NVML for NVIDIA) to identify available accelerators. It then registers these devices with the kubelet via a Unix domain socket at /var/lib/kubelet/device-plugins/. The kubelet advertises this capacity to the API server as an extended resource, typically nvidia.com/gpu, amd.com/gpu, or similar. When a pod specifies this resource in its spec, the scheduler's filter phase eliminates nodes lacking sufficient capacity, and the allocate phase calls back into the device plugin to mount device files (/dev/nvidia*) and inject necessary environment variables into the container runtime.

A common mistake is assuming GPU resources behave like CPU. They do not. GPUs are non-compressible and generally non-shareable by default. If a pod requests one whole GPU but uses only 10% of its VRAM, the remaining 90% sits idle. Understanding this constraint is fundamental before configuring your cluster. For teams managing mixed workloads, reviewing Kubernetes resource limits and requests provides essential context for how extended resources differ from standard compute primitives.

How do you install and configure the NVIDIA GPU Operator?

In 2026, the NVIDIA GPU Operator remains the standard for production clusters. It automates driver installation, container toolkit setup, device plugin deployment, and optional components like DCGM exporter for monitoring. Avoid manual driver installs on Kubernetes nodes; they break during kernel upgrades and node recycling.

Prerequisites and Installation

Ensure your nodes have a compatible Linux kernel, secure boot disabled (or properly signed modules), and no pre-existing NVIDIA drivers. Install the operator via Helm:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update

helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.enabled=true \
  --set toolkit.enabled=true \
  --set dcgmExporter.enabled=true \
  --version=24.9.0

The operator creates a validator pod first. Watch for nvidia-operator-validator to reach Ready status before deploying workloads. If validation fails, check kubectl logs -n gpu-operator nvidia-operator-validator for driver compatibility or kernel header issues. On Ubuntu nodes specifically, ensure you've completed proper Ubuntu security hardening while keeping necessary kernel modules loadable.

Verifying GPU Registration

After installation, confirm GPUs are visible to the scheduler:

kubectl get nodes -o json | jq '.items[] | {name: .metadata.name, gpus: .status.allocatable["nvidia.com/gpu"]}'

You should see integer GPU counts per node. If values are missing or zero, the device plugin has not successfully registered. Check the device plugin daemonset logs and verify the /var/lib/kubelet/device-plugins/kubelet.sock exists on the host.

When should you use MIG versus time-slicing for GPU sharing?

Whole-GPU allocation wastes resources when workloads don't saturate hardware. Two primary sharing strategies exist for GPU scheduling on Kubernetes: Multi-Instance GPU (MIG) and time-slicing. Choosing correctly depends on your workload isolation requirements and hardware generation.

To enable time-slicing with the GPU Operator, create a custom resource:

apiVersion: nvidia.com/v1alpha1
kind: ClusterPolicy
metadata:
  name: gpu-operator-policy
spec:
  devicePlugin:
    config:
      name: time-slicing-config
---
apiVersion: v1
kind: ConfigMap
metadata:
  name: time-slicing-config
  namespace: gpu-operator
data:
  config.yaml: |
    version: v1
    sharing:
      timeSlicing:
        resources:
          - name: nvidia.com/gpu
            replicas: 4

This makes each physical GPU appear as four schedulable units. Pods requesting nvidia.com/gpu: 1 now receive 25% of a GPU's time. Monitor closely: if multiple replicas run simultaneously, latency increases non-linearly. For production serving workloads where tail latency matters, prefer MIG or whole-GPU allocation.

How do you write correct pod specs and node affinity for GPU workloads?

Requesting GPUs incorrectly is the most frequent cause of scheduling failures. Always specify GPU resources under resources.limits, not just requests. Kubernetes enforces that GPU limits equal requests; you cannot overcommit GPUs at the pod level.

apiVersion: v1
kind: Pod
metadata:
  name: llm-inference
spec:
  containers:
    - name: vllm
      image: vllm/vllm-openai:v0.6.3
      resources:
        limits:
          nvidia.com/gpu: 1
          memory: "32Gi"
        requests:
          nvidia.com/gpu: 1
          memory: "32Gi"
      env:
        - name: NVIDIA_VISIBLE_DEVICES
          value: "all"
  nodeSelector:
    nvidia.com/gpu.product: "NVIDIA-H100-80GB-HBM3"
  tolerations:
    - key: nvidia.com/gpu
      operator: Exists
      effect: NoSchedule

Note the nodeSelector. Without it, your pod may land on any GPU node, including older generations incompatible with your workload. Label your GPU nodes during provisioning with instance type, GPU model, and memory size. For large-scale clusters, consider using horizontal pod autoscaling with custom metrics from DCGM exporter to scale inference deployments based on actual GPU utilization rather than CPU or memory proxies.

Always include the nvidia.com/gpu toleration. GPU nodes typically carry a NoSchedule taint to prevent non-GPU workloads from consuming expensive instances. Missing this toleration causes immediate scheduling failure even when GPUs are available.

What observability and troubleshooting practices prevent GPU waste?

GPUs are too valuable to run blind. Deploy the DCGM Exporter (included in GPU Operator) to emit metrics like DCGM_FI_DEV_GPU_UTIL, DCGM_FI_DEV_FB_USED, and DCGM_FI_PROF_PIPE_TENSOR_ACTIVE. Integrate these with your existing Prometheus and Grafana monitoring stack to build dashboards showing real utilization per node, per pod, and per GPU slice.

Common issues and resolutions:

• Pods stuck in Pending: Run kubectl describe pod <name> and check Events. "Insufficient nvidia.com/gpu" means all GPUs are allocated. "Node didn't match node selector" indicates label mismatch. Verify actual node labels with kubectl get nodes --show-labels.
• Container fails with CUDA error: Usually a driver/runtime version mismatch. Ensure the GPU Operator's toolkit version matches your application's CUDA base image. Pin versions explicitly.
• Low utilization despite high allocation: Your workload is CPU-bound or data-loading bound. Profile with nsys or DCGM profiling metrics before adding more GPUs.
• MIG configuration errors: MIG profiles must sum exactly to physical GPU capacity. An A100-80GB supports specific combinations (e.g., seven 1g.10gb slices OR two 3g.40gb + one 1g.10gb). Invalid configurations cause the device plugin to crash-loop.

Set up alerts for sustained low utilization (<30% over 15 minutes) and persistent pending GPU pods. These signals indicate either over-provisioning or capacity constraints requiring cluster scaling. In compliance-sensitive environments, log GPU allocation events as part of your audit trail; SOC 2 auditors increasingly scrutinize expensive resource governance.

Optimizing GPU Scheduling on Kubernetes for Production

Effective GPU scheduling on Kubernetes demands intentional architecture: deploy the GPU Operator instead of manual drivers, choose MIG for isolated multi-tenancy or time-slicing for flexible dev environments, enforce strict pod specs with node affinity and tolerations, and instrument everything with DCGM metrics. Treat GPUs as scarce, stateful infrastructure—not interchangeable compute. Audit allocations regularly, right-size workloads against actual utilization data, and automate scaling decisions based on real GPU signals rather than heuristics. If your team needs help designing a compliant, cost-efficient GPU platform or debugging persistent scheduling issues, reach out to discuss your specific workload requirements.