NVIDIA Container Toolkit for Docker GPU

By Khimananda Oli | Last reviewed: August 2026

Running machine learning workloads or graphics processing inside containers requires direct hardware access that standard Docker cannot provide alone. The NVIDIA Container Toolkit for Docker GPU bridges this gap by injecting necessary driver libraries and device nodes into the container namespace at runtime without baking them into your image. This guide walks you through the exact installation, configuration, and validation steps needed to get reliable GPU acceleration working on Ubuntu hosts in 2026.

How does the NVIDIA Container Toolkit for Docker GPU architecture work?

Understanding the architecture prevents the most common configuration errors I see teams make when setting up GPU infrastructure. Unlike legacy approaches that required mounting host driver paths manually, the modern toolkit uses a pre-start OCI hook mechanism. When you launch a container with GPU requests, the Docker engine invokes the nvidia-container-runtime-hook before the container process starts. This hook queries the host's NVIDIA driver, determines compatible library versions, and bind-mounts only the specific shared objects and device files (/dev/nvidia*) required by that container.

This separation is critical for maintainability. Your container images remain portable and lean because they do not contain host-specific kernel modules. If you are building foundational infrastructure knowledge, understanding these isolation boundaries complements core Linux administration skills covered in resources like Ubuntu for developers guide. The toolkit essentially acts as a secure translation layer between the host kernel and the containerized user space.

How do you install NVIDIA drivers and the toolkit on Ubuntu?

A frequent failure point in GPU setups is mismatched driver versions. Before installing the toolkit, you must have the proprietary NVIDIA driver installed and verified on the host. Nouveau (the open-source driver) does not support the compute APIs required for containerized workloads. On Ubuntu 22.04 and 24.04 LTS, use the official PPA or the built-in ubuntu-drivers utility for the most stable experience.

Step 1: Verify host GPU and driver status

# Check if NVIDIA GPU is detected by the kernel
lspci | grep -i nvidia

# Verify the proprietary driver is loaded
nvidia-smi

# Expected output should show GPU name, driver version, and utilization table
# If this fails, install drivers first:
sudo apt update
sudo ubuntu-drivers autoinstall
sudo reboot

Step 2: Add the NVIDIA container repository

Never install nvidia-docker2 from random DEB files found online. Always use the official upstream repository to receive security patches and compatibility updates for new CUDA releases.

# Add the GPG key and repository (Ubuntu 24.04 example)
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg

curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
  sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
  sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt update

Step 3: Install the toolkit packages

sudo apt install -y nvidia-container-toolkit

Note that in 2026, the meta-package nvidia-docker2 is largely deprecated in favor of directly configuring the runtime via nvidia-container-toolkit. Installing the toolkit alone gives you full control over the daemon configuration without pulling in legacy wrapper scripts.

How do you configure the Docker daemon runtime for GPU access?

Installation alone does not enable GPU passthrough. You must explicitly tell Docker to use the NVIDIA runtime. There are two methods: modifying the daemon JSON or using the CLI tool provided by the toolkit. The daemon JSON method is preferred for production servers because it persists across reboots and integrates cleanly with Infrastructure as Code tools discussed in Infrastructure as Code with Terraform.

Method A: Daemon JSON configuration (Recommended)

Edit or create /etc/docker/daemon.json. If the file already exists, merge the runtimes block carefully; invalid JSON will prevent Docker from starting.

{
    "runtimes": {
        "nvidia": {
            "args": [],
            "path": "nvidia-container-runtime"
        }
    },
    "default-runtime": "nvidia"
}

Setting default-runtime to nvidia means every container gets GPU access by default. For multi-tenant hosts where only specific workloads need GPUs, omit the default-runtime line and specify --runtime=nvidia per container instead.

Method B: Automatic configuration via CLI

# Automatically patches daemon.json and restarts Docker
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Validation checklist after restart

• Docker service is active: systemctl status docker
• Runtime is registered: docker info | grep -i runtime should list nvidia
• No errors in journal: journalctl -u docker --since "5 minutes ago"

How do you run and validate GPU containers in production?

Once configured, you request GPUs using the standard --gpus flag introduced in Docker 19.03+. Avoid the legacy --runtime=nvidia syntax in new deployments; the device request API is more granular and aligns with Kubernetes device plugin standards.

Basic validation command

# Run nvidia-smi inside a minimal CUDA container
docker run --rm --gpus all nvidia/cuda:12.6.0-base-ubuntu24.04 nvidia-smi

If this outputs the GPU table matching your host, the toolkit is functioning correctly. If you see "could not select device driver "" with capabilities: [[gpu]]", revisit the daemon configuration and ensure the Docker service was fully restarted.

Selecting specific GPUs

In multi-GPU servers, you rarely want every container claiming all devices. Use ordinal indices or UUIDs for deterministic scheduling:

# Allocate only GPU 0 and GPU 2
docker run --rm --gpus '"device=0,2"' my-ml-app:latest python train.py

# Allocate by UUID (more stable across reboots/replacements)
docker run --rm --gpus '"device=GPU-abc123..."' my-ml-app:latest

Production considerations for monitoring

GPU workloads generate distinct telemetry signals. Standard CPU/memory metrics won't tell you if your model training is bottlenecked on VRAM bandwidth or compute. Integrate GPU metrics into your observability stack early. Tools like DCGM Exporter feed Prometheus with GPU utilization, temperature, and ECC error counts. For broader context on what signals matter, review the four golden signals of monitoring and adapt them for accelerator-heavy workloads. Without this visibility, you cannot distinguish between a healthy idle GPU and a hung kernel.

NVIDIA Container Toolkit vs legacy nvidia-docker: Which should you use?

Teams migrating older infrastructure often ask whether to keep using the nvidia-docker2 wrapper package. In 2026, the answer is almost always to migrate to the native toolkit. The comparison below clarifies why the ecosystem has moved forward.

Troubleshooting common GPU container failures

Even with correct installation, subtle issues can prevent GPU access. Work through this diagnostic sequence when containers fail to see devices:

1. Check host driver health first: Run nvidia-smi on the host. If it errors, the problem is not Docker-related. Reinstall drivers or check for kernel taints.
2. Verify toolkit installation: Run nvidia-ctk --version. Missing binary indicates incomplete package installation.
3. Inspect container logs for hook errors: Messages like "nvidia-container-cli: mount error" usually indicate SELinux/AppArmor denials or missing device nodes.
4. Confirm cgroup compatibility: Systems using cgroup v2 require toolkit version ≥1.13. Check with stat -fc %T /sys/fs/cgroup (cgroup2fs = v2).
5. Review daemon.json syntax: A trailing comma or malformed JSON silently disables the runtime. Validate with python3 -m json.tool /etc/docker/daemon.json.

In regulated environments, remember that GPU access expands your attack surface. Containers with GPU passthrough can potentially perform side-channel attacks or exhaust hardware resources. Apply least-privilege principles: never grant --gpus all to untrusted workloads, and consider using NVIDIA MPS (Multi-Process Service) for controlled sharing. These practices align with compliance frameworks like SOC 2 where hardware resource isolation is an audit requirement.

Next steps for GPU-enabled infrastructure

Getting the NVIDIA Container Toolkit for Docker GPU working is just the foundation. Reliable production systems need automated provisioning, metric collection, and capacity planning around those accelerators. If you are managing multiple GPU nodes or integrating this into a Kubernetes cluster, evaluate the NVIDIA GPU Operator to handle driver and toolkit lifecycle management declaratively. For teams needing help designing compliant, observable GPU infrastructure, reach out to discuss your architecture. Proper setup now prevents costly debugging and security incidents when your ML workloads scale.