To install NVIDIA drivers on an Ubuntu 24.04 server, add NVIDIA's cuda-keyring package for the ubuntu2404 repository, run apt install nvidia-open, reboot and check nvidia-smi. Then install Docker Engine from Docker's apt repository and the NVIDIA Container Toolkit (1.20.1-1 on 16 November 2026), and prove the stack with docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi. The current driver, 620.70.06, lists both the RTX PRO 6000 Blackwell and the RTX PRO 4000 Blackwell SFF.
This is the first-hour checklist for a MassiveGRID GPU dedicated server that has just been delivered with a clean Ubuntu 24.04 LTS install and root SSH access. Nothing is pre-installed, so the guide covers the whole path: updates, SSH and firewall basics, the driver, Docker, the container toolkit, the optional CUDA Toolkit, the two NVMe drives, and a troubleshooting table.
What is on the server when it arrives, and what this hour adds
A MassiveGRID GPU dedicated server is a bare-metal machine: the whole server and the whole GPU belong to one customer, and you log in as root over SSH on a dedicated IPv4 address. The operating system you chose at order (Ubuntu 24.04 LTS is the default) is clean. There is no NVIDIA driver, no CUDA, no Docker and no Python environment on it, so nvidia-smi does not exist yet. This guide installs each layer in order and checks it before moving on.
| Server | GPU | vRAM | CPU | RAM | NVMe |
|---|---|---|---|---|---|
| RTX PRO 4000, $449.99 per month | RTX PRO 4000 Blackwell SFF Edition | 24 GB GDDR7 ECC | Core i5-13500, 14 cores / 20 threads | 64 GB DDR4 | 2x 512 GB |
| RTX PRO 6000, $1,656.15 per month | RTX PRO 6000 Blackwell Max-Q | 96 GB GDDR7 ECC | Core Ultra 7 265, 20 cores / 20 threads | 128 GB DDR5 | 2x 1.92 TB |
| RTX PRO 6000 Xeon, $2,485.05 per month | RTX PRO 6000 Blackwell Max-Q | 96 GB GDDR7 ECC | Xeon Gold 5412U, 24 cores / 48 threads | 256 GB DDR5 ECC | 2x 960 GB |
| 4x RTX PRO 6000, $5,966.44 per month | 4x RTX PRO 6000 Blackwell Max-Q | 384 GB (4x 96 GB) | Xeon Gold 5412U, 24 cores / 48 threads | 128 GB DDR5 ECC | 2x 1.92 TB |
The steps are the same on all four GPU dedicated server plans. The order matters: kernel first, then the driver, then Docker, then the container toolkit, because each layer is built against the one below it.
- Update the system and reboot into the current kernel.
- Lock down SSH and switch on the firewall.
- Install the NVIDIA driver from NVIDIA's apt repository and reboot.
- Install Docker Engine from Docker's apt repository.
- Install the NVIDIA Container Toolkit and run the verification container.
- Optionally install the CUDA Toolkit on the host.
- Decide what the two NVMe drives do and where backups go.
Commands below are written as the vendors document them. Where NVIDIA's guide shows a root prompt, the command appears here without sudo and you run it as root; where a vendor writes sudo, the command works unchanged from a root shell.
Step 1: update Ubuntu and reboot into the current kernel
The NVIDIA kernel modules are compiled on the server for the kernel that is running. If you install the driver first and a kernel update arrives the same day, the module has to be rebuilt, and that is a common reason for a GPU that vanishes after the first reboot. So update first.
apt update
apt full-upgrade -y
rebootLog back in and confirm which kernel is running. You will need the matching headers in step 3.
uname -r
lsb_release -dsCheck: lsb_release -ds prints Ubuntu 24.04 and apt list --upgradable shows nothing left to upgrade. If you intend to hold a kernel version on a long-running production server, decide that now, before the driver is built.
Step 2: SSH keys only, and a firewall that allows nothing else
The server has a public IPv4 address and a 1 Gbps port, so automated scanners will find it. Two changes cover most of the risk in the first hour: no password logins, and no open port except SSH.
Copy your public key to the server from your own computer (ssh-copy-id root@SERVER_IP), open a second session to prove the key works, and only then disable passwords. Create a drop-in file so a package update does not overwrite it:
cat > /etc/ssh/sshd_config.d/10-hardening.conf <<'EOF'
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-password
EOF
sshd -t
systemctl restart sshsshd -t prints nothing when the configuration is valid. Keep the first session open until a new login succeeds.
Ubuntu ships ufw, which is disabled by default. Allow SSH before enabling it, or you lock yourself out:
sudo ufw allow 22
sudo ufw enable
sudo ufw status verboseCheck: the status output shows the default policy "deny (incoming)" and a rule for port 22. Open 80 and 443 later, when a reverse proxy with TLS sits in front of whatever you serve; never open a model server's own port (11434 for Ollama, 8000 for vLLM, 8188 for ComfyUI) to the internet.
One warning for step 4: Docker's documentation states that when you expose container ports using Docker, "these ports bypass your firewall rules". A container started with -p 8000:8000 is reachable from the internet even though ufw lists the port as closed. Publish container ports on the loopback address (-p 127.0.0.1:8000:8000) and put a reverse proxy in front.
Step 3: install the NVIDIA driver from NVIDIA's apt repository
NVIDIA's Driver Installation Guide documents one route for Ubuntu 22.04, 24.04 and 26.04: add the repository keyring, then install the nvidia-open package, which contains the open kernel modules. On Ubuntu 24.04 for x86_64 the repository name is ubuntu2404:
apt install linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt install nvidia-open
rebootThree notes on that block.
- Which version you get. The repository installs the newest driver branch. On 16 November 2026 the latest Linux x86_64 driver was 620.70.06, and its supported-products list includes the RTX PRO 6000 Blackwell Max-Q Workstation Edition and the RTX PRO 4000 Blackwell SFF Edition, the two cards in these servers.
- Pinning. To stay on one branch, NVIDIA documents a pinning package that you install before the driver:
apt install nvidia-driver-pinning-<branch>(or-<version>). Since branch 590 the Ubuntu package names no longer contain the branch number, so the pinning packages are the supported way to hold a version. - Headless variant. A server has no desktop. For the smallest footprint, NVIDIA documents a compute-only install in place of
nvidia-open:apt -V install libnvidia-compute nvidia-dkms-open. It leaves out the GL, EGL, Vulkan and X components; you can add them later if a rendering workload needs them.
Ubuntu also has its own route (sudo ubuntu-drivers install --gpgpu). The examples in Ubuntu's documentation use the 535 branch, which predates the Blackwell cards, so this guide uses NVIDIA's repository, where the version is known.
After the reboot:
nvidia-smi
cat /proc/driver/nvidia/versionCheck: nvidia-smi prints a table with the driver version, one row per GPU (four rows on the 4x server) and the memory total, which is 24 GB on the RTX PRO 4000 and 96 GB per RTX PRO 6000 (the tool reports it in MiB). If the command is missing or reports that it cannot communicate with the driver, go to the troubleshooting table before continuing. Nothing later in this guide works until this step does.
Step 4: install Docker Engine from Docker's repository
Most AI software is distributed as container images with the matching CUDA libraries inside, which is why the host needs only the driver. Install Docker Engine from Docker's own apt repository, as its Ubuntu guide documents:
# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc
# Add the repository to Apt sources:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF
sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-pluginCheck:
sudo systemctl status docker
sudo docker run hello-worldThe second command pulls a test image and prints a confirmation message. At this point Docker works but containers cannot see the GPU; that is the next step.
Step 5: install the NVIDIA Container Toolkit and run the verification container
The NVIDIA Container Toolkit is the piece that hands the GPU and the driver libraries to a container. Its install guide (version 1.20.1-1 on 16 November 2026) gives these commands for apt-based systems:
sudo apt-get update && sudo apt-get install -y --no-install-recommends \
ca-certificates \
curl \
gnupg2
curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
&& curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list
sudo apt-get update
export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}Then register the NVIDIA runtime with Docker and restart the daemon:
sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart dockerCheck: NVIDIA's sample workload runs nvidia-smi inside a plain Ubuntu container:
sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smiYou should see the same table as on the host. The Ubuntu image contains no NVIDIA software, so a correct table proves that the toolkit injected the driver libraries and the device. From here every GPU container follows the same pattern: --gpus all for every card, or --gpus '"device=0"' for one card on the 4x server.
Optional: install the CUDA Toolkit on the host
You do not need the CUDA Toolkit on the host to run containers, Ollama, or a Python wheel that bundles its own CUDA libraries (vLLM's binaries, for example, are built against CUDA 12.9). Install it only if you compile CUDA code on the host. The keyring from step 3 already points at the right repository, so NVIDIA's CUDA installation guide (written for CUDA 13.5 on 16 November 2026) reduces to one package and one path entry:
apt install cuda-toolkit
export PATH=/usr/local/cuda-13.5/bin${PATH:+:${PATH}}Add the export line to ~/.bashrc to keep it. Check: nvcc --version prints the compiler version. The toolkit and the driver are separate packages: updating one does not replace the other.
The two NVMe drives: RAID, model storage and backups
Each server has two NVMe drives. A dedicated server is a single machine: there is no second copy of your data anywhere unless you make one, so redundancy and backups are your job. Start by looking at how the installer laid the drives out:
lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
cat /proc/mdstatlsblk shows both drives and what is mounted where; /proc/mdstat lists any Linux software RAID arrays and whether they are in sync. Then decide between two layouts:
- RAID 1 (mirror). Both drives hold the same data. Usable space is one drive: 512 GB on the RTX PRO 4000 server, 1.92 TB on the RTX PRO 6000 and 4x servers, 960 GB on the base Xeon configuration. The server keeps running if one drive fails. This is the sensible default for a production inference server.
- Two independent drives (or RAID 0). Usable space is both drives: 1,024 GB, 3.84 TB or 1.92 TB. One drive failure loses what was on it. Reasonable only for data you can download again, such as public model weights.
Model files are large. The default Ollama build of Llama 3.3 70B is a 43 GB download and gpt-oss 120B is 65 GB; the full-precision Llama 3.3 70B repository on Hugging Face is 141.1 GB. On the 512 GB drives of the 24 GB server, plan which models you keep. Ollama stores models in /usr/share/ollama/.ollama/models on Linux (change it with OLLAMA_MODELS) and Hugging Face tools default to ~/.cache/huggingface, so both land on the root filesystem unless you move them.
RAID is not a backup: it copies a deleted file or a bad configuration to both drives at once. Weights can be pulled again, but fine-tuned adapters, vector databases, chat histories, Docker volumes and your configuration cannot. Send those off the server on a schedule, to object storage or to another machine, and restore one file as a test before you rely on it.
What goes wrong in the first hour
| Symptom | Likely cause | What to do |
|---|---|---|
nvidia-smi: command not found after the reboot | The driver package did not install (an interrupted apt run, or the keyring step was skipped) | Run apt update and apt install nvidia-open again and read the output for errors. |
nvidia-smi cannot communicate with the NVIDIA driver | The kernel module was not built for the running kernel: the headers were missing, or a kernel update arrived after the driver | Install linux-headers-$(uname -r), run dkms status, rebuild as NVIDIA's FAQ describes (dkms build, then dkms install, for the listed module and kernel) and reboot. |
The module is built but will not load, and dmesg mentions a rejected key or signature | UEFI Secure Boot is on and the key that signed the module is not enrolled | Check with mokutil --sb-state. Either enrol the DKMS machine owner key, which needs console access during boot, or ask support whether Secure Boot can be switched off for the server. |
lsmod | grep nouveau shows the open-source nouveau driver loaded | nouveau claimed the card before the NVIDIA module | Confirm that the NVIDIA packages are installed and reboot once. If nouveau still loads, rebuild the initramfs with update-initramfs -u and reboot again. |
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]] | Docker does not know the NVIDIA runtime: the Container Toolkit is missing, or Docker was not restarted after it was configured | Repeat step 5: install the toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, then sudo systemctl restart docker. |
| A container that worked now prints "Failed to initialize NVML: Unknown Error" | NVIDIA documents this for containers that lose GPU access, for example after systemctl daemon-reload on hosts where systemd manages the container cgroups | Restart the container. NVIDIA's troubleshooting page lists permanent workarounds, including the cgroupfs cgroup driver for Docker. |
| A port you never opened in ufw answers from the internet | A container was started with -p PORT:PORT, which bypasses ufw | Recreate it with -p 127.0.0.1:PORT:PORT and reach it through a TLS reverse proxy or an SSH tunnel. |
With the driver, Docker and the toolkit verified, the server is ready for a workload. The next guides in this series start from exactly this state: a private chat assistant with Ollama and Open WebUI, an OpenAI-compatible API with vLLM on the 96 GB card, and splitting the RTX PRO 6000 into isolated instances with MIG. To watch temperature, memory and utilisation over time, add the exporter described in GPU monitoring with Prometheus and Grafana. If you are still choosing hardware, the RTX PRO 6000 server page explains what fits in 96 GB.
Frequently Asked Questions
Which NVIDIA driver supports the RTX PRO 6000 Blackwell and RTX PRO 4000 Blackwell on Linux?
Driver 620.70.06, the latest Linux x86_64 release on 16 November 2026, lists the RTX PRO 6000 Blackwell Workstation, Max-Q Workstation and Server Editions and the RTX PRO 4000 Blackwell SFF Edition as supported. Installing nvidia-open from NVIDIA's ubuntu2404 repository gives you the current branch. If you plan to use Multi-Instance GPU on the RTX PRO 6000, NVIDIA requires driver 575.51.03 or later, which the current driver satisfies.
Do I need to install CUDA on the host to run AI containers?
No. The host needs the NVIDIA driver and the NVIDIA Container Toolkit; the container image brings its own CUDA libraries. Install the cuda-toolkit package on the host only if you compile CUDA code there. Ollama and vLLM both run without it.
Should I use nvidia-open or the Ubuntu drivers tool?
NVIDIA's Driver Installation Guide documents nvidia-open, the package with the open kernel modules, for Ubuntu 22.04, 24.04 and 26.04. Ubuntu's ubuntu-drivers tool is the distribution's own route, but the examples in its documentation use the 535 branch, which is older than the Blackwell cards. On a new GPU, NVIDIA's repository is the route where you know which version you get.
Is the GPU server delivered with drivers or Docker pre-installed?
No. A MassiveGRID GPU dedicated server arrives with the operating system you picked at order and root access, and you install the rest. The steps on this page take a clean Ubuntu 24.04 install to a verified GPU container. Other choices at order include Ubuntu 26.04 and 22.04, Debian, AlmaLinux and Rocky Linux; NVIDIA's guide has a page for each distribution family.
Does RAID 1 on the two NVMe drives replace backups?
No. A mirror protects against one drive failing and nothing else: a deleted file, a corrupted database or a compromised server is mirrored too. Keep copies of fine-tuned adapters, databases, Docker volumes and configuration somewhere off the server, and test a restore.
GPU dedicated servers from $449.99 per month
MassiveGRID GPU dedicated servers run NVIDIA RTX PRO Blackwell cards on hardware that is yours alone: one RTX PRO 4000 with 24 GB for $449.99 per month, one RTX PRO 6000 with 96 GB from $1,656.15, or four RTX PRO 6000 with 384 GB for $5,966.44. Every server has a 1 Gbps port with unlimited traffic, a dedicated IPv4 address, root access and a choice of Ubuntu, Debian, AlmaLinux or Rocky Linux, and can be ordered online.
See the GPU ServersFurther Reading
- NVIDIA Driver Installation Guide: Ubuntu 22.04, 24.04 and 26.04
- NVIDIA driver 620.70.06: supported products
- Docker Docs: Install Docker Engine on Ubuntu (apt repository, firewall limitations)
- NVIDIA Container Toolkit: installation guide
- NVIDIA Container Toolkit: running a sample workload
- NVIDIA CUDA Installation Guide for Linux
- Ubuntu Server documentation: firewalls (ufw)
- NVIDIA Container Toolkit: troubleshooting