NVIDIA GPU Operator for Kubernetes

By Khimananda Oli | Last reviewed: August 2026

Running AI or HPC workloads on bare metal requires more than just installing drivers; you need a consistent, automated way to manage GPU lifecycles across dynamic nodes. The NVIDIA GPU Operator for Kubernetes solves this by automating the provisioning of drivers, container runtimes, and device plugins via Helm, eliminating manual node configuration. If you are building an Amazon EKS cluster or managing on-prem infrastructure, understanding this operator is essential for reliable GPU scheduling.

How does the NVIDIA GPU Operator for Kubernetes work?

The operator replaces the legacy approach of baking GPU drivers into base AMIs or running manual installation scripts on every node. Instead, it uses a declarative model where the desired state of GPU software is defined in a Custom Resource Definition (CRD). When you apply the operator, it deploys a DaemonSet that detects GPU hardware and orchestrates the entire software stack as containers.

This architecture provides three critical advantages over traditional methods. First, it decouples the GPU software stack from the host OS, allowing you to upgrade drivers independently of kernel updates. Second, it ensures consistency; every node runs the exact same validated software versions regardless of when it joined the cluster. Third, it integrates directly with Kubernetes primitives, making GPU resources visible to the scheduler immediately after node readiness. For teams managing custom operators, this pattern demonstrates how complex hardware dependencies can be abstracted into manageable API objects.

How do you install and configure the NVIDIA GPU Operator?

Installation is straightforward via Helm, but configuration requires attention to your specific environment. Before starting, ensure your nodes have a supported Linux distribution and that the open-source kernel headers match your running kernel. The operator will fail silently if it cannot compile or load the driver module against the current kernel version.

Step-by-step Helm installation

1. Add the NVIDIA Helm repository and update your local cache:

   helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
   helm repo update

2. Create a namespace for GPU operator resources to maintain isolation:

   kubectl create namespace gpu-operator

3. Install the operator with default settings for most environments:

   helm install --wait --generate-name \
     -n gpu-operator \
     nvidia/gpu-operator

4. Verify all pods reach Running state before scheduling workloads:

   kubectl get pods -n gpu-operator -w

In practice, default values rarely suffice for production. You will likely need a custom values.yaml to specify driver versions, enable MIG, or configure private registries. Always pin the operator chart version and driver version explicitly; floating tags like "latest" are unacceptable in audited environments where reproducibility matters for compliance and debugging.

When should you enable Multi-Instance GPU (MIG) strategy?

MIG partitions a single physical GPU into up to seven isolated instances, each with dedicated compute, memory, and cache. This is valuable when running multiple smaller inference workloads that would otherwise waste GPU capacity. However, MIG adds complexity and is not suitable for all scenarios. Understanding the trade-offs prevents costly misconfigurations.

Enable MIG by setting mig.strategy=mixed or mig.strategy=single in your Helm values. The mixed strategy allows different MIG profiles on the same node, offering maximum flexibility at the cost of scheduler complexity. The single strategy enforces uniform partitioning across all GPUs on a node, simplifying resource accounting. For teams new to MIG, start with single and validate workload performance before attempting mixed configurations. Remember that enabling MIG requires a node reboot and re-partitioning, so plan maintenance windows accordingly.

How do you troubleshoot common GPU Operator failures?

Even with automation, things break. The most frequent issues stem from kernel mismatches, network restrictions, or resource contention. A systematic debugging approach saves hours of guessing. Start by checking the operator pod logs and node conditions before diving into driver internals.

• Driver container CrashLoopBackOff: Usually indicates missing kernel headers or incompatible kernel version. Verify kernel-devel package matches uname -r exactly. Check kubectl logs <driver-pod> for compilation errors.
• Device plugin not registering: Often caused by container runtime misconfiguration. Ensure nvidia-container-runtime is set as default in /etc/containerd/config.toml or Docker daemon config. Restart containerd after changes.
• Pods stuck in Pending: Scheduler cannot find available GPU resources. Run kubectl describe node <gpu-node> to verify nvidia.com/gpu capacity is advertised. If zero, device plugin failed initialization.
• MIG partitioning fails: GPU may already be in use or MIG mode disabled in BIOS/firmware. Drain node completely before re-partitioning. Check nvidia-smi mig -l output for current state.
• Pull errors in air-gapped environments: Operator tries to fetch images from NGC by default. Override all image repositories in values.yaml to point to your internal registry. Pre-pull images during node provisioning to avoid runtime delays.

For persistent issues, enable verbose logging by setting operator.defaultRuntime=containerd and toolkit.env[0].name=CONTAINERD_CONFIG in your Helm values. Cross-reference with the official CrashLoopBackOff debugging guide for general pod failure patterns. Remember that GPU operator pods run with elevated privileges; always review RBAC bindings and security contexts when troubleshooting in regulated environments.

What are the best practices for production GPU clusters?

Production GPU infrastructure demands discipline beyond basic installation. Treat GPU nodes as specialized resources with distinct lifecycle management. Implement node taints and tolerations to prevent non-GPU workloads from consuming expensive hardware. Use nvidia.com/gpu=true:NoSchedule taints on GPU nodes and add corresponding tolerations only to ML/AI workloads.

Monitor GPU utilization rigorously. Idle GPUs burn budget without delivering value. Deploy DCGM exporter alongside the operator to expose metrics like DCGM_FI_DEV_GPU_UTIL and DCGM_FI_DEV_FB_USED to Prometheus. Set up alerts for sustained low utilization (<20% for >1 hour) to identify orphaned workloads or over-provisioned jobs. Integrate these signals into your existing monitoring fundamentals stack for unified observability.

Version control your operator configuration. Store values.yaml in Git alongside your infrastructure code. Use ArgoCD or Flux to manage deployments declaratively. This ensures audit trails for compliance and enables rapid rollback if a driver update causes regressions. Never apply GPU operator changes directly via Helm CLI in production; drift between declared and actual state is the enemy of reliability.

Implementing NVIDIA GPU Operator for Kubernetes successfully

The NVIDIA GPU Operator for Kubernetes transforms GPU management from a fragile, manual process into a resilient, automated system. Success depends on treating it as infrastructure code: version pinning, declarative configuration, comprehensive monitoring, and disciplined change management. Start with standard mode, validate your workloads, then explore MIG only when utilization data justifies the complexity. If you need help designing a GPU-ready cluster or auditing your existing setup, reach out to discuss your infrastructure requirements.