CUDA and Container GPU Basics

By Khimananda Oli | Last reviewed: August 2026

Running AI workloads in production requires more than just installing drivers; mastering CUDA and Container GPU Basics is the difference between a model that trains reliably and one that fails silently at 3 AM. Many teams struggle because they treat GPU access as an afterthought rather than a first-class infrastructure dependency. This guide covers the exact configuration patterns I use to deploy stable GPU-accelerated containers on Docker and Kubernetes.

How do you configure the NVIDIA Container Toolkit for Docker?

The most common mistake in CUDA and Container GPU Basics is trying to install GPU drivers inside the container image. Never do this. The container should only contain user-space CUDA libraries; the kernel driver must remain on the host. The NVIDIA Container Toolkit acts as the translation layer, injecting the necessary device nodes and libraries at runtime. For teams building AI infrastructure, getting this separation right is foundational, much like understanding GPUs for AI development before scaling.

Installation and Runtime Configuration

On Ubuntu 24.04 or RHEL 9 systems, install the toolkit using the official repository. Avoid legacy nvidia-docker2 packages; use the modern nvidia-container-toolkit.

# Add NVIDIA package repositories
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg
curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

# Install and configure
sudo apt-get update && sudo apt-get install -y nvidia-container-toolkit
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

After installation, verify the runtime is registered. Running docker info | grep -i runtime should list nvidia. Test access with a minimal CUDA sample:

docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi

If this command fails, check /var/log/nvidia-container-runtime.log. In my experience, 90% of failures here stem from mismatched host driver versions or SELinux/AppArmor blocking device injection. Always validate host drivers with nvidia-smi on the bare metal host before debugging container issues.

What CUDA version matches your host driver?

Version compatibility is where most GPU container deployments break. CUDA forward compatibility allows newer CUDA toolkits to run on older drivers, but only within specific bounds. Understanding these matrices prevents "could not select device driver" errors during deployment. This alignment is as critical as setting proper resource limits in orchestration platforms.

A practical rule: standardize on Long Term Support (LTS) driver branches (e.g., 535 or 550) for production clusters. Chasing the latest CUDA version often forces premature driver upgrades that destabilize multi-tenant nodes. Use the nvidia-smi output's top-right corner to confirm your host's maximum supported CUDA version before selecting a base image tag.

How does Kubernetes schedule GPU workloads reliably?

In Kubernetes, GPUs are extended resources managed by the NVIDIA GPU Operator or device plugin. Unlike CPU/memory, GPUs cannot be oversubscribed by default (though time-slicing is possible). Proper scheduling requires explicit resource requests and node selectors. Teams migrating from single-node Docker setups often miss these orchestration nuances, leading to pods stuck in Pending state. Refer to Amazon EKS practical guide for managed cluster specifics.

Device Plugin and Resource Requests

Install the NVIDIA GPU Operator via Helm to automate device plugin, container runtime, and validator deployment:

helm repo add nvidia https://helm.ngc.nvidia.com/nvidia
helm repo update
helm install gpu-operator nvidia/gpu-operator \
  --namespace gpu-operator --create-namespace \
  --set driver.enabled=false \  # If using pre-installed host drivers
  --set toolkit.enabled=true

In your pod spec, request GPUs as nvidia.com/gpu. Never use generic gpu keys unless you've configured custom resource mappings:

resources:
  limits:
    nvidia.com/gpu: 1  # Must equal requests for GPUs
  requests:
    nvidia.com/gpu: 1
nodeSelector:
  nvidia.com/gpu.product: "NVIDIA-A10G"

A frequent pitfall: setting limits without matching requests. Kubernetes rejects GPU pods where these values differ. Also, avoid requesting fractional GPUs unless you've explicitly enabled time-slicing in the operator config. For multi-instance GPU (MIG) strategies on A100/H100 hardware, define separate resource types like nvidia.com/mig-1g.5gb to enable secure tenant isolation.

Why are my containerized GPU workloads underperforming?

Performance issues in GPU containers usually trace back to three root causes: improper memory pinning, missing persistence mode, or thermal throttling. Debugging requires systematic validation beyond simple nvidia-smi checks. Monitoring these signals aligns with observability best practices for high-value compute resources.

• Persistence Mode: Without it, the GPU driver unloads between jobs, causing 2-5 second initialization latency. Enable globally with sudo nvidia-smi -pm 1 or via systemd service.
• Memory Pinning: In NUMA-aware systems, ensure containers bind to the same NUMA node as their assigned GPU. Use --cpuset-cpus and --cpuset-mems in Docker or topology-aware scheduling in K8s.
• ECC Memory: For training workloads, verify ECC is enabled (nvidia-smi -q | grep ECC). Disabled ECC risks silent data corruption during long runs.
• Power Limits: Cloud instances often cap GPU power below TDP. Check nvidia-smi -q -d POWER and adjust if permitted by your provider.

For persistent storage bottlenecks, ensure dataset volumes use local NVMe or high-throughput network storage. GPU starvation from slow I/O is indistinguishable from compute issues without proper metrics. Always instrument data loading pipelines separately from model execution.

Production Checklist for GPU Infrastructure

Mastering CUDA and Container GPU Basics means treating GPU access as a disciplined engineering practice, not a magic configuration. Before promoting any AI workload to production, verify host driver stability, pin exact container base image tags, enable persistence mode, and validate NUMA topology. Automate these checks in your CI pipeline or node provisioning scripts to prevent drift. If your team needs help designing audit-ready GPU infrastructure or optimizing existing AI deployments, reach out to discuss your architecture.