With vLLM on a multi-GPU server you choose between two layouts: --tensor-parallel-size 4 spreads one model across all four cards, and four separate vllm serve processes, each pinned to a card with CUDA_VISIBLE_DEVICES, serve four replicas behind a load balancer. On four RTX PRO 6000 cards (384 GB) the first layout runs models that no single card holds: qwen3:235b is a 142 GB download, llama3.1:405b 243 GB, llama4:maverick 245 GB and qwen3-coder:480b 290 GB at 4-bit. vLLM's guidance is to split a model only when it does not fit on one GPU.
This guide is for the MassiveGRID 4x RTX PRO 6000 server after a clean Ubuntu 24.04 install: which models need all four cards and which still do not fit, the commands for both layouts, how Ollama distributes a model on its own, how to plan around 128 GB of system RAM and two 1.92 TB drives, and how to watch each GPU.
One big model or four replicas: how to decide
The 4x RTX PRO 6000 server is one machine with four 96 GB cards, a Xeon Gold 5412U (24 cores, 48 threads), 128 GB of DDR5 ECC memory and two 1.92 TB NVMe drives, for $5,966.44 per month. Over a 730-hour month that is $8.17 per hour, or $2.04 per GPU-hour. vLLM's guidance picks between three ways to use the 384 GB: if the model fits on a single GPU, do not distribute it; if it is too large for one GPU but fits on one node, use tensor parallelism.
| Layout | vLLM setting | Use it when | What you give up |
|---|---|---|---|
| One model across four cards | --tensor-parallel-size 4 | The weights are larger than 192 GB, or you need a very large cache for long contexts | All four cards serve one model; every request involves all of them |
| Two models, or two replicas, on two cards each | --tensor-parallel-size 2 with CUDA_VISIBLE_DEVICES=0,1 and 2,3 | The weights are between about 96 GB and 192 GB | Half the server per instance |
| Four independent replicas | One vllm serve per card, CUDA_VISIBLE_DEVICES=0 to 3, a load balancer in front | The model fits in 96 GB and you want throughput, or four different models | No model larger than one card |
Split a model across cards only because it does not fit, never to make a small model faster. Everything that fits one card is covered in the single-GPU vLLM guide for the 96 GB card; this guide starts where that one ends.
Which models need four cards, and what still does not fit?
Two size columns matter, because the two serving stacks use different files. Ollama pulls GGUF builds, 4-bit (Q4_K_M) by default. vLLM loads Hugging Face repositories, where the original is usually 16-bit and the practical choice at this size is a 4-bit AWQ build. Sizes are from the Ollama library and the Hugging Face API on 16 November 2026.
| Model | Ollama default tag | Larger builds | Cards needed (96 GB each) | Fits in 384 GB? |
|---|---|---|---|---|
| Qwen3 235B (MoE, 22B active) | qwen3:235b, 142 GB | q8: 250 GB; BF16 on Hugging Face: 470.2 GB; AWQ 4-bit: 124.1 GB | 2 for the 4-bit builds, 3 to 4 for q8 | Yes at 4-bit and q8; no at BF16 |
| Llama 4 Maverick | llama4:maverick, 245 GB | q8: 428 GB | 3 to 4 | Yes at 4-bit; no at q8 |
| Llama 3.1 405B | llama3.1:405b, 243 GB | q8: 431 GB; fp16: 812 GB | 3 to 4 | Yes at 4-bit only |
| Qwen3-Coder 480B (MoE, 35B active) | qwen3-coder:480b, 290 GB | q8: 510 GB; AWQ 4-bit on Hugging Face: 252.3 GB | 4 | Yes at 4-bit only |
| DeepSeek-R1 671B, DeepSeek V3.1 | deepseek-r1:671b, 404 GB | q8: 713 GB; Hugging Face FP8: 688.6 GB | More than 4 | No |
Weights are the floor; the key-value cache for every active conversation comes on top. A 290 GB model leaves 94 GB across four cards for cache, which suits chat-length contexts and not very long ones. The Qwen3-235B model card says a 1 million token context needs "approximately 1000 GB of total GPU memory" and advises reducing the context to 32,768 tokens on out-of-memory errors.
A 142 GB or 124.1 GB build of Qwen3 235B needs only two cards, so the server can run two copies of it. The largest open models stay out of reach: a 404 GB download is larger than 384 GB of VRAM. Check licences too: the Qwen models are Apache 2.0, while Llama 4 and Llama 3.1 carry Meta's community licences.
Option A: one model across four GPUs with tensor parallel size 4
Install vLLM 0.33.0 as in the single-GPU guide (a uv virtual environment, uv pip install vllm --torch-backend=auto). The driver and nvidia-smi showing four cards are prerequisites; the first-hour setup guide covers them. vLLM's documented multi-GPU example is vllm serve facebook/opt-13b --tensor-parallel-size 4. With a model that needs the cards, an API key and a loopback bind, it becomes:
export VLLM_API_KEY=$(openssl rand -hex 32)
vllm serve QuantTrio/Qwen3-Coder-480B-A35B-Instruct-AWQ \
--tensor-parallel-size 4 \
--host 127.0.0.1 \
--port 8000 \
--max-model-len 32768This is the 252.3 GB AWQ build that CloudRift ran on four RTX PRO 6000 cards with tensor parallelism in its November 2025 benchmark; CloudRift put its VRAM requirement at approximately 320 GB. Tensor parallelism divides the weights evenly: 252.3 GB over four cards is about 63 GB per card, and the rest of each card is cache.
Check:
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv
curl http://localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"All four cards should show similar memory use, and the API should list the model. If loading fails with out of memory, lower --max-model-len first.
Two cards per model. For a model between one and two cards in size, restrict the process to a pair of GPUs and halve the parallelism. Two copies of the 124.1 GB AWQ build of Qwen3 235B, about 62 GB of weights per card:
CUDA_VISIBLE_DEVICES=0,1 vllm serve QuantTrio/Qwen3-235B-A22B-Instruct-2507-AWQ \
--tensor-parallel-size 2 --host 127.0.0.1 --port 8001 --max-model-len 32768
CUDA_VISIBLE_DEVICES=2,3 vllm serve QuantTrio/Qwen3-235B-A22B-Instruct-2507-AWQ \
--tensor-parallel-size 2 --host 127.0.0.1 --port 8002 --max-model-len 32768vLLM also has --pipeline-parallel-size, which splits a model by layers; CloudRift used it for its multi-GPU RTX tests in October 2025 and tensor parallelism in November 2025. If throughput matters, run vllm bench serve against both on your own model.
Option B: four replicas behind a load balancer
When the model fits on one card, start one server per card. CUDA_VISIBLE_DEVICES decides which card a process sees; each replica gets its own port on the loopback interface. A systemd template unit keeps the four identical:
cat > /etc/vllm.env <<'EOF'
VLLM_API_KEY=paste-the-key-here
EOF
chmod 600 /etc/vllm.env
cat > /etc/systemd/system/vllm@.service <<'EOF'
[Unit]
Description=vLLM replica on GPU %i
After=network-online.target
Wants=network-online.target
[Service]
User=vllm
Group=vllm
WorkingDirectory=/home/vllm
EnvironmentFile=/etc/vllm.env
Environment=CUDA_VISIBLE_DEVICES=%i
ExecStart=/home/vllm/.venv/bin/vllm serve Qwen/Qwen3.8-27B --host 127.0.0.1 --port 800%i --max-model-len 32768
Restart=always
RestartSec=10
[Install]
WantedBy=multi-user.target
EOF
systemctl daemon-reload
systemctl enable --now vllm@0 vllm@1 vllm@2 vllm@3The instance name is the GPU index, so vllm@2 runs on GPU 2 and listens on port 8002. Start the first one alone the first time, so the 55.6 GB download happens once and the other three find it in the cache.
Check: nvidia-smi shows four processes, one per card, and each of curl http://localhost:8000/health to 8003 returns 200.
Put a load balancer in front. Caddy does TLS and balancing in one file; its reverse_proxy directive accepts several upstreams, a balancing policy (the default is random; least_conn picks the upstream with the fewest current requests) and an active health check. In /etc/caddy/Caddyfile:
llm.example.com {
reverse_proxy :8000 :8001 :8002 :8003 {
lb_policy least_conn
health_uri /health
}
}sudo ufw allow 80
sudo ufw allow 443
sudo systemctl reload caddyleast_conn suits LLM traffic, where requests last from a second to minutes. A replica that is still loading its model fails the health check and receives no traffic. CloudRift's benchmarks used the same replica design with NGINX as the balancer.
Security: the replicas listen on 127.0.0.1 only; ports 8000 to 8003 are never opened in ufw; the only public entry is HTTPS on 443 with the API key required.
How does Ollama spread a model across several GPUs?
Ollama needs no flag for multiple GPUs. Its FAQ describes the behaviour: "If the model will entirely fit on any single GPU, Ollama will load the model on that GPU... If the model does not fit entirely on one GPU, then it will be spread across all the available GPUs." So ollama pull qwen3:235b followed by a prompt loads the 142 GB model across the cards without further configuration.
ollama pull qwen3:235b
ollama run qwen3:235b "ready"
ollama ps
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csvCheck: ollama ps shows 100% GPU and the memory query shows which cards hold the model. Three settings matter on this server:
- Context length. On systems with 48 GiB of VRAM or more, Ollama's context-length page gives a default of 256k tokens. Set
OLLAMA_CONTEXT_LENGTHto what you need; on a 235B model the default is a large allocation. - Loaded models.
OLLAMA_MAX_LOADED_MODELSdefaults to 3 times the number of GPUs, 12 here. Ollama's FAQ adds that with GPU inference a new model must fit completely in VRAM to be loaded alongside the others. - Restricting cards. Ollama's GPU documentation names
CUDA_VISIBLE_DEVICESfor limiting which GPUs it uses. Setting it to0,1in the service override leaves cards 2 and 3 free for vLLM.
Ollama is the quick way to try a 142 GB or 290 GB model; for many concurrent users, vLLM's batching justifies the extra setup. The Ollama and Open WebUI guide shows where the environment variables go.
System RAM, downloads and storage: plan before you pull
RAM. The server has 384 GB of VRAM and 128 GB of system memory, so the largest models are bigger than RAM. Two consequences. A 243 GB or 290 GB weight file cannot sit in the page cache, so every cold start reads it from NVMe again; keep models loaded (OLLAMA_KEEP_ALIVE=-1, or a vLLM service that stays up) instead of loading on demand. And there is no useful fallback: a model that does not fit in VRAM would run partly on the CPU from system RAM, far slower, so treat 384 GB as a hard limit.
Storage. Two 1.92 TB drives give 1.92 TB usable as a RAID 1 mirror, or 3.84 TB as independent drives with no protection against a drive failure. Add up what you plan to keep:
| Set of models | Disk space |
|---|---|
| The four Ollama tags above (142 + 245 + 243 + 290 GB) | 920 GB |
| Both AWQ repositories for vLLM (124.1 + 252.3 GB) | 376.4 GB |
| One BF16 original for your own quantisation: Qwen3-235B | 470.2 GB |
| All of the above | 1,766.6 GB, 92% of a 1.92 TB mirror |
Ollama and Hugging Face keep separate caches (/usr/share/ollama/.ollama/models and ~/.cache/huggingface; move them with OLLAMA_MODELS and HF_HOME). Public weights can be downloaded again, but a dedicated server is a single machine: fine-tuned adapters, datasets and logs you must keep need a copy off the server.
Downloads. The port is 1 Gbps with unlimited traffic, so pulling models costs nothing extra, but it takes time. At the full line rate of 125 MB per second, 290 GB takes at least 39 minutes and 470.2 GB at least 63 minutes; the real rate depends on the far end.
Monitoring each GPU
With four cards an average hides the problem, so watch them individually. From the shell:
nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw --format=csv -l 5For history and alerts, NVIDIA's DCGM exporter publishes per-GPU metrics for Prometheus on port 9400. Its documented run command, with the port bound to the loopback interface so that it is not exposed:
export DCGM_EXPORTER_TAG=latest
docker run -d --rm \
--name dcgm-exporter \
--gpus all \
--cap-add SYS_ADMIN \
-p 127.0.0.1:9400:9400 \
nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}
curl --fail http://localhost:9400/metricsCheck: the output contains metric lines starting with DCGM_, each labelled with a GPU index. Each vLLM replica also exposes Prometheus metrics on /metrics. The Grafana dashboard guide for GPU servers turns both sources into panels.
Published multi-GPU results, and where other hardware wins
MassiveGRID does not publish benchmark figures for this server. CloudRift has published two vLLM studies on RTX PRO 6000 cards that show what to expect from each layout (checked on 16 November 2026; its cards were the Workstation Edition where stated, these servers carry the Max-Q edition):
- Replicas scale, but not perfectly. With GLM-4.5-Air AWQ 4-bit, one RTX PRO 6000 produced 3,140 output tokens per second and eight replicas behind NGINX produced 15,713 in total, against 25,120 for perfect scaling. CloudRift calls that about 63% scaling efficiency and attributes it to load balancer overhead, queuing and batch-size choices (27 November 2025).
- One model over four cards is where data center GPUs pull ahead. On Qwen3-Coder-480B AWQ with tensor parallelism across four GPUs, CloudRift reported the H100 about 31% ahead of the RTX PRO 6000 in throughput, with cost per million tokens nearly tied on its price list. On a larger FP8 model spread across eight GPUs, the H100 was nearly three times as fast.
For models that fit one card, four replicas on this server are a strong layout: CloudRift measured a single RTX PRO 6000 slightly ahead of a single H100 on such a model. For one very large model across all cards, and for large-scale training, H100 and H200 class systems are the better tool; MassiveGRID quotes those on the H100, H200 and B200 page. For reference, four of the same cards on demand for a 730-hour month were $13,139.80 at Google Cloud (g4-standard-192, Iowa), $12,097.68 at AWS (g7e.24xlarge, us-east-1) and $8,760.00 at Akamai (listed at $12.00 an hour) on 16 November 2026, so the break-even against this server is about 331, 360 and 497 hours a month. Below that many hours a month, paying by the hour costs less, and at low volume a pay-per-token API gives access to frontier models without a server. The four-card server earns its $5,966.44 when it is busy most of the day, when data has to stay on hardware that is yours alone, or when you need 384 GB at a fixed price with unlimited traffic. The GPU dedicated server plans and the multi-GPU server page compare it with the single-card servers.
What goes wrong on a multi-GPU server
| Symptom | Cause | Fix |
|---|---|---|
| Out of memory while loading with tensor parallelism | Weights fit, but the cache for the requested context does not | Lower --max-model-len (32,768 is a sensible start for the largest models). |
| Only one card fills up, then the load fails | --tensor-parallel-size missing, or CUDA_VISIBLE_DEVICES restricts the process to one card | Check the unit file and the environment; run nvidia-smi during the load. |
Ollama shows a CPU percentage in ollama ps | Model plus context exceeds the VRAM that is free, often because another model is still loaded | Lower OLLAMA_CONTEXT_LENGTH, unload other models with ollama stop, or use a smaller build. |
| Cold starts take many minutes | Hundreds of gigabytes are read from disk on every load | Keep the model loaded; do not restart services casually. |
| Clients get errors from the balancer while replicas load | No health check, so traffic reaches a replica that is not ready | Add health_uri /health to the Caddy block. |
Frequently Asked Questions
What does --tensor-parallel-size do in vLLM?
It splits one model across several GPUs in the same server, so that a model too large for one card can be served. The default is 1. vLLM's documentation advises using it only when the model does not fit on a single GPU but fits on one node, for example --tensor-parallel-size 4 on a node with four GPUs. For a model that fits one card, separate replicas give more throughput.
How much VRAM do I need to run a 235B model?
For Qwen3 235B it depends on the build: the default Ollama tag is a 142 GB download, an AWQ 4-bit repository for vLLM is 124.1 GB, the q8 build is 250 GB and the original BF16 weights are 470.2 GB. The 4-bit builds fit on two 96 GB cards with room for context, q8 needs three to four, and BF16 does not fit in 384 GB. Add memory for the context on top of the weights.
Can I run four different models on a four-GPU server?
Yes. Start one server process per card with CUDA_VISIBLE_DEVICES=0 to 3 and a different port each, and route by hostname or path in the reverse proxy. Each model must fit in 96 GB with its cache. You can also mix layouts, for example one model on two cards with --tensor-parallel-size 2 and two single-card models.
Does Ollama use all four GPUs automatically?
Only when it has to. Ollama's FAQ says a model that fits entirely on one GPU is loaded on that GPU, and a model that does not fit on one is spread across all available GPUs. Use CUDA_VISIBLE_DEVICES in the service environment to limit which cards it may use.
What does the 4x RTX PRO 6000 server cost per GPU?
The server is $5,966.44 per month with a one-time setup fee of the same amount, which is $1,491.61 per card per month, $2.04 per GPU-hour over a 730-hour month, and $15.54 per GB of VRAM per month. It includes a Xeon Gold 5412U, 128 GB of DDR5 ECC, 2x 1.92 TB NVMe, a 1 Gbps port with unlimited traffic and a dedicated IPv4 address. It is listed with the single-card servers on the GPU dedicated servers page.
Four RTX PRO 6000 cards, 384 GB, one tenant
The MassiveGRID 4x RTX PRO 6000 server is bare metal with four 96 GB RTX PRO 6000 Blackwell cards, a Xeon Gold 5412U, 128 GB of DDR5 ECC and 2x 1.92 TB NVMe for $5,966.44 per month ($2.04 per GPU-hour over a 730-hour month). It has a 1 Gbps port with unlimited traffic, a dedicated IPv4 address and root access, hosted in an ISO 27001 certified data center.
See Multi-GPU ServersFurther Reading
- vLLM documentation: parallelism and scaling (tensor and pipeline parallelism)
- vLLM documentation: engine arguments
- Ollama FAQ: how models load on multiple GPUs, concurrency settings
- Ollama documentation: context length defaults by VRAM
- Hugging Face: Qwen3-235B-A22B-Instruct-2507 model card (context and memory notes)
- CloudRift: RTX PRO 6000 vs data center GPUs (27 November 2025)
- Caddy: reverse_proxy directive (load balancing and health checks)
- NVIDIA DCGM exporter: installation and run command
- AWS EC2 on-demand pricing (checked on 16 November 2026)
- Google Cloud accelerator-optimized machine pricing (g4-standard-192)
- Akamai Cloud GPU pricing