Four 96 GB GPUs in One Dedicated Server
A multi-GPU server puts several cards in one machine, so that one large model can span them or several models can run side by side. The 4x NVIDIA RTX PRO 6000 server has 384 GB of vRAM (4x 96 GB), a 24-core Xeon Gold 5412U and 128 GB of RAM for $5,966.44 a month, which works out at $8.17 an hour or $2.04 per GPU-hour over a 730-hour month. It is the largest of the four servers on the GPU dedicated servers page.
What is a multi-GPU server, and what do four 96 GB cards do?
Multi-GPU server: one machine with two or more graphics cards that software can use together or separately. On this server the four cards are NVIDIA RTX PRO 6000 Blackwell Max-Q Workstation Edition boards, each with 96 GB of GDDR7 memory with ECC and a 300 W board power limit (NVIDIA datasheet, checked on 16 November 2026). There are three ways to use them.
| Mode | What it gives you | How |
|---|---|---|
| One model across four cards | A single model with up to 384 GB of weights, context and cache | Tensor parallelism in vLLM: --tensor-parallel-size 4 |
| Four independent models | Four 96 GB endpoints, for example chat, code, embeddings and vision, or four replicas of one model | One process per card, pinned with CUDA_VISIBLE_DEVICES |
| Up to 16 isolated slices | 16 instances of 24 GB, or 8 of 48 GB, each with its own memory and compute | NVIDIA Multi-Instance GPU (MIG): up to four instances per card |
vLLM's own guidance decides between the first two: if a model fits on one GPU, do not distribute it, and if it does not fit on one GPU but fits on one node, use tensor parallelism. A model that fits in 96 GB therefore runs better as four replicas than as one split copy. CloudRift published that pattern on 27 November 2025: eight RTX PRO 6000 cards running eight replicas of GLM-4.5-Air (AWQ 4-bit) reached 15,713 output tokens per second in total against 3,140 for one card. Those were the 600 W Workstation Edition cards, not the 300 W Max-Q cards in this server, so measure your own model before you size a service on that figure.
MIG has conditions that NVIDIA documents for the Max-Q card: driver branch R575 (575.51.03) or later, vBIOS 98.02.6A.00.00 or later, and a switch from graphics mode to compute mode with NVIDIA's DisplayModeSelector tool before the first use. MIG mode does not persist across a reboot or driver reload, so the instances are created again by a boot script. The profiles are 1g.24gb (four per card), 2g.48gb (two per card) and 4g.96gb (the whole card). The MIG guide for the RTX PRO 6000 has the commands.
Which models need more than 96 GB and fit in 384 GB?
The sizes below are download sizes as the Ollama library and Hugging Face list them, checked on 16 November 2026. They are weights only: the context window and the KV cache come on top, and they grow with context length and with the number of parallel requests. A model is listed as fitting when the weights leave room for that.
| Model and precision | Weights | One 96 GB card | Four cards, 384 GB |
|---|---|---|---|
| Llama 3.3 70B, Q4_K_M / q8_0 | 43 GB / 75 GB | Fits | Fits, run replicas |
| Llama 3.3 70B, fp16 (BF16 on Hugging Face: 141.1 GB) | 141 GB | Does not fit | Fits, about 243 GB left |
| gpt-oss 120b, MXFP4 | 65 GB | Fits | Fits, run replicas |
| Qwen3 235B (22B active), Q4_K_M / q8_0 | 142 GB / 250 GB | Does not fit | Fits |
| Mistral Medium 3.5 128B, q8_0 / bf16 | 138 GB / 255 GB | Does not fit (Q4_K_M at 80 GB does) | Fits |
| Devstral 2 123B, q8_0 / fp16 | 133 GB / 250 GB | Does not fit (Q4_K_M at 75 GB does) | Fits |
| Llama 4 Scout (109B), q8_0 / fp16 | 117 GB / 217 GB | Does not fit (Q4_K_M at 67 GB does) | Fits |
| Llama 3.1 405B, Q4_K_M | 243 GB | Does not fit | Fits, about 141 GB left |
| Qwen3-Coder 480B (35B active), Q4_K_M | 290 GB | Does not fit | Weights fit, 94 GB left for context |
| Qwen3 235B, fp16 | 470 GB | Does not fit | Does not fit |
| DeepSeek-R1 671B, Q4_K_M (FP8 on Hugging Face: 688.6 GB) | 404 GB | Does not fit | Does not fit |
| Mistral Large 3 675B / Mistral Large 4 | 681.5 GB / 1,060.6 GB | Does not fit | Does not fit |
Check the licence before you build on a model: Qwen3 and gpt-oss are Apache 2.0, the Llama models carry Meta's community licences, and Devstral 2 and Mistral Medium 3.5 use a modified MIT licence with restrictions for large-revenue companies. If your model sits in the first rows only, one card is enough: the single RTX PRO 6000 server costs $1,656.15 a month, and the LLM vRAM requirements guide explains how context adds to the weights.
How much GPU memory does fine-tuning need?
Unsloth publishes minimum vRAM per model size for its two adapter methods: QLoRA loads the base model in 4-bit, LoRA in 16-bit. It calls the numbers "the absolute minimum" (read on 16 November 2026); batch size and sequence length add to them.
| Model size | QLoRA, 4-bit | LoRA, 16-bit | Smallest server by memory |
|---|---|---|---|
| 14B | 8.5 GB | 33 GB | 24 GB card for QLoRA, $449.99; 96 GB for LoRA |
| 27B | 22 GB | 64 GB | One 96 GB card, $1,656.15 |
| 32B | 26 GB | 76 GB | One 96 GB card |
| 40B | 30 GB | 96 GB | One card for QLoRA; LoRA needs more than one |
| 70B | 41 GB | 164 GB | One card for QLoRA; four cards for LoRA, $5,966.44 |
| 90B | 53 GB | 212 GB | One card for QLoRA; four cards for LoRA |
| 405B | 237 GB | 950 GB | Four cards for QLoRA; LoRA does not fit |
Full fine-tuning is a different order of size. LLaMA-Factory's estimates for full 16-bit training are 120 GB for a 14B model, 300 GB for 30B and 600 GB for 70B, so 384 GB covers full training up to roughly the 30B class by that estimate and not a 70B model. Memory is the total across cards: whether a job actually spreads over four GPUs depends on the training framework, so read its multi-GPU documentation before ordering. For the single-card workflow, see fine-tuning an LLM on a single GPU.
What are the limits of this server?
Four things to know before you order, because they decide whether this machine or a quoted system is right.
- 128 GB of system RAM for 384 GB of vRAM. The host has a third as much memory as the cards. AWS pairs the same four cards with 1,024 GiB in its g7e.24xlarge. Data loading, tokenisation and anything else that lives in host memory has to fit in 128 GB, and this configuration lists no RAM upgrade.
- A 1 Gbps port. Traffic is unlimited, but a 243 GB model takes over half an hour to download at the full line rate (243 GB is 1,944 gigabits, so at least 1,944 seconds), and datasets take as long as their size dictates.
- Workstation Max-Q cards. They are the 300 W blower edition of the RTX PRO 6000, not the 600 W Workstation Edition and not the Server Edition. The third-party throughput figures on this page were measured on 600 W cards. We publish no throughput figure for the Max-Q card: measure it with your model.
- No interconnect claim. The store listing gives no details about how the four cards are linked, so this page makes none. Tensor-parallel scaling depends on the model, so benchmark yours before you commit to a long term.
It is also a single machine. Two NVMe drives of 1.92 TB let you run software RAID 1, but redundancy and off-server backups of checkpoints and adapters are your job.
What does a GPU-hour cost here and on hourly multi-GPU clouds?
The server is billed monthly, not hourly. Divided by 730 hours, $5,966.44 is $8.17 an hour, or $2.04 for each of the four GPUs. The cloud figures below are on-demand list prices for the same RTX PRO 6000 Blackwell card, checked on 16 November 2026, before storage and egress.
| Offer | GPUs and host | Per hour, on demand | Per GPU-hour | 730 hours | Hours at which $5,966.44 is less |
|---|---|---|---|---|---|
| MassiveGRID 4x RTX PRO 6000 (monthly) | 4x 96 GB, 48 threads, 128 GB | $8.17 equivalent | $2.04 | $5,966.44 | - |
| AWS g7e.24xlarge, Frankfurt | 4x 96 GB, 96 vCPU, 1,024 GiB | $28.18177 | $7.05 | $20,572.69 | About 212 |
| Google Cloud g4-standard-192, Iowa | 4x 96 GB, 192 vCPU, 720 GiB | $17.99972 | $4.50 | $13,139.80 | About 331 |
| AWS g7e.24xlarge, N. Virginia | 4x 96 GB, 96 vCPU, 1,024 GiB | $16.57216 | $4.14 | $12,097.68 | About 360 |
| Akamai Cloud RTX PRO 6000 Blackwell x4 (as listed) | 4x 96 GB, 64 vCPU, 704 GB | $12.00 | $3.00 | $8,760.00 | About 497 |
| Latitude.sh g4.rtx6kpro.large, bare metal (as listed) | 8x 96 GB | $24.00 | $3.00 | $17,520.00 | About 497, against two servers |
| CoreWeave 8x RTX PRO 6000 Blackwell | 8x 96 GB, 128 vCPU, 1,024 GB | $20.00 | $2.50 | $14,600.00 | About 597, against two servers |
Over a full month the four-card instances cost about 2 times the server's price on AWS in N. Virginia, about 2.2 times on Google Cloud in Iowa and about 1.5 times on Akamai, and per GPU the eight-card offers from Latitude.sh and CoreWeave come to about 1.5 and 1.2 times. Azure's two-card NC288ds is $11.00 an hour in East US, $5.50 per GPU-hour. On a monthly contract, Vultr's public API lists a bare-metal server with four A100 cards at $7,000 a month, with no location shown on 16 November 2026. Egress is extra on the hyperscalers: $0.09 per GB on AWS and $0.12 per GiB on Google's Premium Tier.
Where hourly cloud fits. Below the hours in the last column, paying by the hour costs less, so rent by the hour for short or bursty jobs such as a single fine-tuning run. The first month here is $11,932.88 because the one-time setup fee equals one month. The AWS and Google instances also carry far more system RAM (1,024 GiB and 720 GiB against 128 GB).
Where the dedicated server wins. A fixed bill for a service that runs all month, unlimited traffic on the port, the whole machine for one tenant with root access, and billing terms from one month to three years. The GPU cloud alternative page has the full price table for single-GPU instances.
One multi-GPU server, one step down, one step up
Four cards and 384 GB to order online, a single 96 GB card with up to 768 GB of system RAM, or H100, H200 and B200 systems by quote.
4x NVIDIA RTX PRO 6000
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
NVIDIA RTX PRO 6000 Xeon
- 3,511 AI TOPS
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
NVIDIA H100 / H200 / B200
Every RTX PRO server includes: a 1 Gbps port with unlimited traffic, one dedicated IPv4 address, hosting in an ISO 27001 certified data center, and your choice of Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS Stream, openSUSE or Arch Linux, or a rescue system to install your own OS. Billing runs monthly, quarterly, semi-annually, annually, biennially or triennially, and the setup fee is charged once with the first invoice.
Options on the RTX PRO 6000 Xeon: 512 GB DDR5 ECC with 2x 1.92 TB NVMe for $994.68/mo more, 768 GB DDR5 ECC with 4x 3.84 TB NVMe for $1,989.37/mo more, and a 10 Gbps uplink with 20 TB of traffic included for $71.29/mo more.
How to bring up four GPUs and serve one model across them
The server arrives with a clean operating system (Ubuntu 24.04 LTS by default) and root access. Nothing is pre-installed, so the first hour is drivers, then a serving stack. The commands are NVIDIA's and vLLM's documented ones.
- Install the NVIDIA driver from NVIDIA's apt repository and reboot:
apt install linux-headers-$(uname -r) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb dpkg -i cuda-keyring_1.1-1_all.deb apt update apt install nvidia-open reboot - Verify that all four cards are visible.
nvidia-smimust list four RTX PRO 6000 devices with 96 GB each. If it lists fewer, stop and open a support ticket before you install anything else. - Install vLLM in a virtual environment:
uv venv --python 3.12 --seed --managed-python source .venv/bin/activate uv pip install vllm --torch-backend=auto - Serve one model across the four cards with an API key, then check the endpoint:
vllm serve <model> --tensor-parallel-size 4 --api-key <your-key> curl http://localhost:8000/v1/models -H "Authorization: Bearer <your-key>" - Measure. Run your own prompts at the concurrency you expect and record tokens per second and time to first token. If the model fits on one card, compare four replicas against the split copy.
vLLM listens on port 8000. Keep it behind a firewall or a TLS reverse proxy rather than exposing it directly. The step-by-step version is in multi-GPU vLLM on four RTX PRO 6000 cards and the first-hour driver guide.
When are H100, H200 or B200 systems the better choice?
Four RTX PRO 6000 cards suit inference and adapter fine-tuning of models up to a few hundred gigabytes. Three cases point to data-center cards instead, which MassiveGRID configures and prices by quote:
- The model does not fit in 384 GB. DeepSeek-R1 at Q4_K_M is a 404 GB download. Eight H200 cards hold 1,128 GB and eight B200 cards 1,440 GB. Mistral's card for Mistral Large 3 says the FP8 weights run "on a single node of B200s or H200s".
- Full training, not adapters. LLaMA-Factory estimates 600 GB for full 16-bit fine-tuning of a 70B model, and large-scale training also wants more system RAM than 128 GB.
- Throughput per card on large split models. CloudRift reported on 27 November 2025 that four H100 cards ran Qwen3-Coder-480B (AWQ) about 31% faster than four RTX PRO 6000 cards, and that eight H100 cards ran GLM-4.6 in FP8 about three times faster than eight RTX PRO 6000 cards. On a model that fits one card the two were close: 3,140 tokens per second for the RTX PRO 6000 against 2,987 for the H100 with GLM-4.5-Air.
The H100, H200 and B200 page lists the specifications and what to send sales, or go straight to contact us.
Other GPU server pages
One card, a different workload, or a different way to buy.
RTX PRO 6000 Server
One 96 GB card for $1,656.15 a month, or with a Xeon and up to 768 GB of RAM.
LLMLLM Hosting
Which open-weight model fits which server, from 24 GB to 384 GB.
InferenceAI Inference Server
Serving an OpenAI-compatible API from your own hardware.
By quoteH100, H200 and B200 Systems
Data-center cards with 80 GB to 180 GB each, configured by sales.
CostGPU Cloud Alternative
Break-even hours against AWS, Azure, Google Cloud and Oracle on-demand prices.
PrivacyPrivate AI Hosting
Open-weight models on hardware one tenant controls.
Multi-GPU guides
vLLM Multi-GPU on Four RTX PRO 6000 Cards: Tensor Parallelism, Replicas and 235B Models
vLLM multi GPU on four 96 GB cards: --tensor-parallel-size 4 for one model up to 384 GB, or four replicas behind a load balancer. What a 235B model needs.
GuideNVIDIA MIG on the RTX PRO 6000: Split One 96 GB GPU into Isolated Instances
NVIDIA MIG on the RTX PRO 6000 Blackwell splits 96 GB into four 24 GB or two 48 GB instances. Prerequisites, nvidia-smi commands and Docker device selection.
GuidevLLM on One 96 GB GPU: Serving Llama 3.3 70B, Qwen, gpt-oss-120b and DeepSeek Distills
vLLM serve Llama 3.3 70B on one 96 GB GPU: the 141.1 GB original does not fit, a 39.8 GB AWQ build does. Flags, API key, curl test, systemd unit.
GuideFine-Tuning an Open Model on a Single GPU Server
Fine-tune an LLM on one GPU: why LoRA and QLoRA fit, sequence length as the real limit, a working PEFT config, data quality, when to stop, and merge vs adapter.
GuideRTX PRO 6000 Blackwell vs H100 vs RTX 5090
RTX PRO 6000 Blackwell vs H100 vs RTX 5090: 96 GB, 80 GB and 32 GB compared on specs, vLLM benchmarks, licence terms and rental prices on 16 November 2026.
GuideGPU Server Pricing in 2026: Dedicated Monthly Servers vs Hourly Cloud
GPU server pricing in 2026: on-demand hourly and 730-hour cost at AWS, Azure, Google Cloud, Oracle and others on 16 November 2026, with break-even hours.
Frequently Asked Questions
--tensor-parallel-size 4, and Ollama spreads a model across all available GPUs when it does not fit on one. vLLM advises against splitting a model that fits on a single GPU: run replicas instead. See the multi-GPU vLLM guide.Order four 96 GB GPUs in one server
384 GB of vRAM for $5,966.44 a month, bare metal with root access. For H100, H200 or B200 systems, request a quote.