To install NVIDIA drivers on an Ubuntu 24.04 server, add NVIDIA's cuda-keyring package for the ubuntu2404 repository, run apt install nvidia-open, reboot and check nvidia-smi. Then install Docker Engine from Docker's apt repository and the NVIDIA Container Toolkit (1.20.1-1 on 16 November 2026), and prove the stack with docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi. The current driver, 620.70.06, lists both the RTX PRO 6000 Blackwell and the RTX PRO 4000 Blackwell SFF.

This is the first-hour checklist for a MassiveGRID GPU dedicated server that has just been delivered with a clean Ubuntu 24.04 LTS install and root SSH access. Nothing is pre-installed, so the guide covers the whole path: updates, SSH and firewall basics, the driver, Docker, the container toolkit, the optional CUDA Toolkit, the two NVMe drives, and a troubleshooting table.

What is on the server when it arrives, and what this hour adds

A MassiveGRID GPU dedicated server is a bare-metal machine: the whole server and the whole GPU belong to one customer, and you log in as root over SSH on a dedicated IPv4 address. The operating system you chose at order (Ubuntu 24.04 LTS is the default) is clean. There is no NVIDIA driver, no CUDA, no Docker and no Python environment on it, so nvidia-smi does not exist yet. This guide installs each layer in order and checks it before moving on.

ServerGPUvRAMCPURAMNVMe
RTX PRO 4000, $449.99 per monthRTX PRO 4000 Blackwell SFF Edition24 GB GDDR7 ECCCore i5-13500, 14 cores / 20 threads64 GB DDR42x 512 GB
RTX PRO 6000, $1,656.15 per monthRTX PRO 6000 Blackwell Max-Q96 GB GDDR7 ECCCore Ultra 7 265, 20 cores / 20 threads128 GB DDR52x 1.92 TB
RTX PRO 6000 Xeon, $2,485.05 per monthRTX PRO 6000 Blackwell Max-Q96 GB GDDR7 ECCXeon Gold 5412U, 24 cores / 48 threads256 GB DDR5 ECC2x 960 GB
4x RTX PRO 6000, $5,966.44 per month4x RTX PRO 6000 Blackwell Max-Q384 GB (4x 96 GB)Xeon Gold 5412U, 24 cores / 48 threads128 GB DDR5 ECC2x 1.92 TB

The steps are the same on all four GPU dedicated server plans. The order matters: kernel first, then the driver, then Docker, then the container toolkit, because each layer is built against the one below it.

  1. Update the system and reboot into the current kernel.
  2. Lock down SSH and switch on the firewall.
  3. Install the NVIDIA driver from NVIDIA's apt repository and reboot.
  4. Install Docker Engine from Docker's apt repository.
  5. Install the NVIDIA Container Toolkit and run the verification container.
  6. Optionally install the CUDA Toolkit on the host.
  7. Decide what the two NVMe drives do and where backups go.

Commands below are written as the vendors document them. Where NVIDIA's guide shows a root prompt, the command appears here without sudo and you run it as root; where a vendor writes sudo, the command works unchanged from a root shell.

Step 1: update Ubuntu and reboot into the current kernel

The NVIDIA kernel modules are compiled on the server for the kernel that is running. If you install the driver first and a kernel update arrives the same day, the module has to be rebuilt, and that is a common reason for a GPU that vanishes after the first reboot. So update first.

apt update
apt full-upgrade -y
reboot

Log back in and confirm which kernel is running. You will need the matching headers in step 3.

uname -r
lsb_release -ds

Check: lsb_release -ds prints Ubuntu 24.04 and apt list --upgradable shows nothing left to upgrade. If you intend to hold a kernel version on a long-running production server, decide that now, before the driver is built.

Step 2: SSH keys only, and a firewall that allows nothing else

The server has a public IPv4 address and a 1 Gbps port, so automated scanners will find it. Two changes cover most of the risk in the first hour: no password logins, and no open port except SSH.

Copy your public key to the server from your own computer (ssh-copy-id root@SERVER_IP), open a second session to prove the key works, and only then disable passwords. Create a drop-in file so a package update does not overwrite it:

cat > /etc/ssh/sshd_config.d/10-hardening.conf <<'EOF'
PasswordAuthentication no
KbdInteractiveAuthentication no
PermitRootLogin prohibit-password
EOF
sshd -t
systemctl restart ssh

sshd -t prints nothing when the configuration is valid. Keep the first session open until a new login succeeds.

Ubuntu ships ufw, which is disabled by default. Allow SSH before enabling it, or you lock yourself out:

sudo ufw allow 22
sudo ufw enable
sudo ufw status verbose

Check: the status output shows the default policy "deny (incoming)" and a rule for port 22. Open 80 and 443 later, when a reverse proxy with TLS sits in front of whatever you serve; never open a model server's own port (11434 for Ollama, 8000 for vLLM, 8188 for ComfyUI) to the internet.

One warning for step 4: Docker's documentation states that when you expose container ports using Docker, "these ports bypass your firewall rules". A container started with -p 8000:8000 is reachable from the internet even though ufw lists the port as closed. Publish container ports on the loopback address (-p 127.0.0.1:8000:8000) and put a reverse proxy in front.

Step 3: install the NVIDIA driver from NVIDIA's apt repository

NVIDIA's Driver Installation Guide documents one route for Ubuntu 22.04, 24.04 and 26.04: add the repository keyring, then install the nvidia-open package, which contains the open kernel modules. On Ubuntu 24.04 for x86_64 the repository name is ubuntu2404:

apt install linux-headers-$(uname -r)
wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb
dpkg -i cuda-keyring_1.1-1_all.deb
apt update
apt install nvidia-open
reboot

Three notes on that block.

Ubuntu also has its own route (sudo ubuntu-drivers install --gpgpu). The examples in Ubuntu's documentation use the 535 branch, which predates the Blackwell cards, so this guide uses NVIDIA's repository, where the version is known.

After the reboot:

nvidia-smi
cat /proc/driver/nvidia/version

Check: nvidia-smi prints a table with the driver version, one row per GPU (four rows on the 4x server) and the memory total, which is 24 GB on the RTX PRO 4000 and 96 GB per RTX PRO 6000 (the tool reports it in MiB). If the command is missing or reports that it cannot communicate with the driver, go to the troubleshooting table before continuing. Nothing later in this guide works until this step does.

Step 4: install Docker Engine from Docker's repository

Most AI software is distributed as container images with the matching CUDA libraries inside, which is why the host needs only the driver. Install Docker Engine from Docker's own apt repository, as its Ubuntu guide documents:

# Add Docker's official GPG key:
sudo apt update
sudo apt install ca-certificates curl
sudo install -m 0755 -d /etc/apt/keyrings
sudo curl -fsSL https://download.docker.com/linux/ubuntu/gpg -o /etc/apt/keyrings/docker.asc
sudo chmod a+r /etc/apt/keyrings/docker.asc

# Add the repository to Apt sources:
sudo tee /etc/apt/sources.list.d/docker.sources <<EOF
Types: deb
URIs: https://download.docker.com/linux/ubuntu
Suites: $(. /etc/os-release && echo "${UBUNTU_CODENAME:-$VERSION_CODENAME}")
Components: stable
Architectures: $(dpkg --print-architecture)
Signed-By: /etc/apt/keyrings/docker.asc
EOF

sudo apt update
sudo apt install docker-ce docker-ce-cli containerd.io docker-buildx-plugin docker-compose-plugin

Check:

sudo systemctl status docker
sudo docker run hello-world

The second command pulls a test image and prints a confirmation message. At this point Docker works but containers cannot see the GPU; that is the next step.

Step 5: install the NVIDIA Container Toolkit and run the verification container

The NVIDIA Container Toolkit is the piece that hands the GPU and the driver libraries to a container. Its install guide (version 1.20.1-1 on 16 November 2026) gives these commands for apt-based systems:

sudo apt-get update && sudo apt-get install -y --no-install-recommends \
   ca-certificates \
   curl \
   gnupg2

curl -fsSL https://nvidia.github.io/libnvidia-container/gpgkey | sudo gpg --dearmor -o /usr/share/keyrings/nvidia-container-toolkit-keyring.gpg \
  && curl -s -L https://nvidia.github.io/libnvidia-container/stable/deb/nvidia-container-toolkit.list | \
    sed 's#deb https://#deb [signed-by=/usr/share/keyrings/nvidia-container-toolkit-keyring.gpg] https://#g' | \
    sudo tee /etc/apt/sources.list.d/nvidia-container-toolkit.list

sudo apt-get update

export NVIDIA_CONTAINER_TOOLKIT_VERSION=1.20.1-1
sudo apt-get install -y \
    nvidia-container-toolkit=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    nvidia-container-toolkit-base=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container-tools=${NVIDIA_CONTAINER_TOOLKIT_VERSION} \
    libnvidia-container1=${NVIDIA_CONTAINER_TOOLKIT_VERSION}

Then register the NVIDIA runtime with Docker and restart the daemon:

sudo nvidia-ctk runtime configure --runtime=docker
sudo systemctl restart docker

Check: NVIDIA's sample workload runs nvidia-smi inside a plain Ubuntu container:

sudo docker run --rm --runtime=nvidia --gpus all ubuntu nvidia-smi

You should see the same table as on the host. The Ubuntu image contains no NVIDIA software, so a correct table proves that the toolkit injected the driver libraries and the device. From here every GPU container follows the same pattern: --gpus all for every card, or --gpus '"device=0"' for one card on the 4x server.

Optional: install the CUDA Toolkit on the host

You do not need the CUDA Toolkit on the host to run containers, Ollama, or a Python wheel that bundles its own CUDA libraries (vLLM's binaries, for example, are built against CUDA 12.9). Install it only if you compile CUDA code on the host. The keyring from step 3 already points at the right repository, so NVIDIA's CUDA installation guide (written for CUDA 13.5 on 16 November 2026) reduces to one package and one path entry:

apt install cuda-toolkit
export PATH=/usr/local/cuda-13.5/bin${PATH:+:${PATH}}

Add the export line to ~/.bashrc to keep it. Check: nvcc --version prints the compiler version. The toolkit and the driver are separate packages: updating one does not replace the other.

The two NVMe drives: RAID, model storage and backups

Each server has two NVMe drives. A dedicated server is a single machine: there is no second copy of your data anywhere unless you make one, so redundancy and backups are your job. Start by looking at how the installer laid the drives out:

lsblk -o NAME,SIZE,TYPE,FSTYPE,MOUNTPOINT
cat /proc/mdstat

lsblk shows both drives and what is mounted where; /proc/mdstat lists any Linux software RAID arrays and whether they are in sync. Then decide between two layouts:

Model files are large. The default Ollama build of Llama 3.3 70B is a 43 GB download and gpt-oss 120B is 65 GB; the full-precision Llama 3.3 70B repository on Hugging Face is 141.1 GB. On the 512 GB drives of the 24 GB server, plan which models you keep. Ollama stores models in /usr/share/ollama/.ollama/models on Linux (change it with OLLAMA_MODELS) and Hugging Face tools default to ~/.cache/huggingface, so both land on the root filesystem unless you move them.

RAID is not a backup: it copies a deleted file or a bad configuration to both drives at once. Weights can be pulled again, but fine-tuned adapters, vector databases, chat histories, Docker volumes and your configuration cannot. Send those off the server on a schedule, to object storage or to another machine, and restore one file as a test before you rely on it.

What goes wrong in the first hour

SymptomLikely causeWhat to do
nvidia-smi: command not found after the rebootThe driver package did not install (an interrupted apt run, or the keyring step was skipped)Run apt update and apt install nvidia-open again and read the output for errors.
nvidia-smi cannot communicate with the NVIDIA driverThe kernel module was not built for the running kernel: the headers were missing, or a kernel update arrived after the driverInstall linux-headers-$(uname -r), run dkms status, rebuild as NVIDIA's FAQ describes (dkms build, then dkms install, for the listed module and kernel) and reboot.
The module is built but will not load, and dmesg mentions a rejected key or signatureUEFI Secure Boot is on and the key that signed the module is not enrolledCheck with mokutil --sb-state. Either enrol the DKMS machine owner key, which needs console access during boot, or ask support whether Secure Boot can be switched off for the server.
lsmod | grep nouveau shows the open-source nouveau driver loadednouveau claimed the card before the NVIDIA moduleConfirm that the NVIDIA packages are installed and reboot once. If nouveau still loads, rebuild the initramfs with update-initramfs -u and reboot again.
docker: Error response from daemon: could not select device driver "" with capabilities: [[gpu]]Docker does not know the NVIDIA runtime: the Container Toolkit is missing, or Docker was not restarted after it was configuredRepeat step 5: install the toolkit, run sudo nvidia-ctk runtime configure --runtime=docker, then sudo systemctl restart docker.
A container that worked now prints "Failed to initialize NVML: Unknown Error"NVIDIA documents this for containers that lose GPU access, for example after systemctl daemon-reload on hosts where systemd manages the container cgroupsRestart the container. NVIDIA's troubleshooting page lists permanent workarounds, including the cgroupfs cgroup driver for Docker.
A port you never opened in ufw answers from the internetA container was started with -p PORT:PORT, which bypasses ufwRecreate it with -p 127.0.0.1:PORT:PORT and reach it through a TLS reverse proxy or an SSH tunnel.

With the driver, Docker and the toolkit verified, the server is ready for a workload. The next guides in this series start from exactly this state: a private chat assistant with Ollama and Open WebUI, an OpenAI-compatible API with vLLM on the 96 GB card, and splitting the RTX PRO 6000 into isolated instances with MIG. To watch temperature, memory and utilisation over time, add the exporter described in GPU monitoring with Prometheus and Grafana. If you are still choosing hardware, the RTX PRO 6000 server page explains what fits in 96 GB.

Frequently Asked Questions

Which NVIDIA driver supports the RTX PRO 6000 Blackwell and RTX PRO 4000 Blackwell on Linux?

Driver 620.70.06, the latest Linux x86_64 release on 16 November 2026, lists the RTX PRO 6000 Blackwell Workstation, Max-Q Workstation and Server Editions and the RTX PRO 4000 Blackwell SFF Edition as supported. Installing nvidia-open from NVIDIA's ubuntu2404 repository gives you the current branch. If you plan to use Multi-Instance GPU on the RTX PRO 6000, NVIDIA requires driver 575.51.03 or later, which the current driver satisfies.

Do I need to install CUDA on the host to run AI containers?

No. The host needs the NVIDIA driver and the NVIDIA Container Toolkit; the container image brings its own CUDA libraries. Install the cuda-toolkit package on the host only if you compile CUDA code there. Ollama and vLLM both run without it.

Should I use nvidia-open or the Ubuntu drivers tool?

NVIDIA's Driver Installation Guide documents nvidia-open, the package with the open kernel modules, for Ubuntu 22.04, 24.04 and 26.04. Ubuntu's ubuntu-drivers tool is the distribution's own route, but the examples in its documentation use the 535 branch, which is older than the Blackwell cards. On a new GPU, NVIDIA's repository is the route where you know which version you get.

Is the GPU server delivered with drivers or Docker pre-installed?

No. A MassiveGRID GPU dedicated server arrives with the operating system you picked at order and root access, and you install the rest. The steps on this page take a clean Ubuntu 24.04 install to a verified GPU container. Other choices at order include Ubuntu 26.04 and 22.04, Debian, AlmaLinux and Rocky Linux; NVIDIA's guide has a page for each distribution family.

Does RAID 1 on the two NVMe drives replace backups?

No. A mirror protects against one drive failing and nothing else: a deleted file, a corrupted database or a compromised server is mirrored too. Keep copies of fine-tuned adapters, databases, Docker volumes and configuration somewhere off the server, and test a restore.

GPU dedicated servers from $449.99 per month

MassiveGRID GPU dedicated servers run NVIDIA RTX PRO Blackwell cards on hardware that is yours alone: one RTX PRO 4000 with 24 GB for $449.99 per month, one RTX PRO 6000 with 96 GB from $1,656.15, or four RTX PRO 6000 with 384 GB for $5,966.44. Every server has a 1 Gbps port with unlimited traffic, a dedicated IPv4 address, root access and a choice of Ubuntu, Debian, AlmaLinux or Rocky Linux, and can be ordered online.

See the GPU Servers

Further Reading