With vLLM on a multi-GPU server you choose between two layouts: --tensor-parallel-size 4 spreads one model across all four cards, and four separate vllm serve processes, each pinned to a card with CUDA_VISIBLE_DEVICES, serve four replicas behind a load balancer. On four RTX PRO 6000 cards (384 GB) the first layout runs models that no single card holds: qwen3:235b is a 142 GB download, llama3.1:405b 243 GB, llama4:maverick 245 GB and qwen3-coder:480b 290 GB at 4-bit. vLLM's guidance is to split a model only when it does not fit on one GPU.

This guide is for the MassiveGRID 4x RTX PRO 6000 server after a clean Ubuntu 24.04 install: which models need all four cards and which still do not fit, the commands for both layouts, how Ollama distributes a model on its own, how to plan around 128 GB of system RAM and two 1.92 TB drives, and how to watch each GPU.

One big model or four replicas: how to decide

The 4x RTX PRO 6000 server is one machine with four 96 GB cards, a Xeon Gold 5412U (24 cores, 48 threads), 128 GB of DDR5 ECC memory and two 1.92 TB NVMe drives, for $5,966.44 per month. Over a 730-hour month that is $8.17 per hour, or $2.04 per GPU-hour. vLLM's guidance picks between three ways to use the 384 GB: if the model fits on a single GPU, do not distribute it; if it is too large for one GPU but fits on one node, use tensor parallelism.

LayoutvLLM settingUse it whenWhat you give up
One model across four cards--tensor-parallel-size 4The weights are larger than 192 GB, or you need a very large cache for long contextsAll four cards serve one model; every request involves all of them
Two models, or two replicas, on two cards each--tensor-parallel-size 2 with CUDA_VISIBLE_DEVICES=0,1 and 2,3The weights are between about 96 GB and 192 GBHalf the server per instance
Four independent replicasOne vllm serve per card, CUDA_VISIBLE_DEVICES=0 to 3, a load balancer in frontThe model fits in 96 GB and you want throughput, or four different modelsNo model larger than one card

Split a model across cards only because it does not fit, never to make a small model faster. Everything that fits one card is covered in the single-GPU vLLM guide for the 96 GB card; this guide starts where that one ends.

Which models need four cards, and what still does not fit?

Two size columns matter, because the two serving stacks use different files. Ollama pulls GGUF builds, 4-bit (Q4_K_M) by default. vLLM loads Hugging Face repositories, where the original is usually 16-bit and the practical choice at this size is a 4-bit AWQ build. Sizes are from the Ollama library and the Hugging Face API on 16 November 2026.

ModelOllama default tagLarger buildsCards needed (96 GB each)Fits in 384 GB?
Qwen3 235B (MoE, 22B active)qwen3:235b, 142 GBq8: 250 GB; BF16 on Hugging Face: 470.2 GB; AWQ 4-bit: 124.1 GB2 for the 4-bit builds, 3 to 4 for q8Yes at 4-bit and q8; no at BF16
Llama 4 Maverickllama4:maverick, 245 GBq8: 428 GB3 to 4Yes at 4-bit; no at q8
Llama 3.1 405Bllama3.1:405b, 243 GBq8: 431 GB; fp16: 812 GB3 to 4Yes at 4-bit only
Qwen3-Coder 480B (MoE, 35B active)qwen3-coder:480b, 290 GBq8: 510 GB; AWQ 4-bit on Hugging Face: 252.3 GB4Yes at 4-bit only
DeepSeek-R1 671B, DeepSeek V3.1deepseek-r1:671b, 404 GBq8: 713 GB; Hugging Face FP8: 688.6 GBMore than 4No

Weights are the floor; the key-value cache for every active conversation comes on top. A 290 GB model leaves 94 GB across four cards for cache, which suits chat-length contexts and not very long ones. The Qwen3-235B model card says a 1 million token context needs "approximately 1000 GB of total GPU memory" and advises reducing the context to 32,768 tokens on out-of-memory errors.

A 142 GB or 124.1 GB build of Qwen3 235B needs only two cards, so the server can run two copies of it. The largest open models stay out of reach: a 404 GB download is larger than 384 GB of VRAM. Check licences too: the Qwen models are Apache 2.0, while Llama 4 and Llama 3.1 carry Meta's community licences.

Option A: one model across four GPUs with tensor parallel size 4

Install vLLM 0.33.0 as in the single-GPU guide (a uv virtual environment, uv pip install vllm --torch-backend=auto). The driver and nvidia-smi showing four cards are prerequisites; the first-hour setup guide covers them. vLLM's documented multi-GPU example is vllm serve facebook/opt-13b --tensor-parallel-size 4. With a model that needs the cards, an API key and a loopback bind, it becomes:

export VLLM_API_KEY=$(openssl rand -hex 32)

vllm serve QuantTrio/Qwen3-Coder-480B-A35B-Instruct-AWQ \
    --tensor-parallel-size 4 \
    --host 127.0.0.1 \
    --port 8000 \
    --max-model-len 32768

This is the 252.3 GB AWQ build that CloudRift ran on four RTX PRO 6000 cards with tensor parallelism in its November 2025 benchmark; CloudRift put its VRAM requirement at approximately 320 GB. Tensor parallelism divides the weights evenly: 252.3 GB over four cards is about 63 GB per card, and the rest of each card is cache.

Check:

nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv
curl http://localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"

All four cards should show similar memory use, and the API should list the model. If loading fails with out of memory, lower --max-model-len first.

Two cards per model. For a model between one and two cards in size, restrict the process to a pair of GPUs and halve the parallelism. Two copies of the 124.1 GB AWQ build of Qwen3 235B, about 62 GB of weights per card:

CUDA_VISIBLE_DEVICES=0,1 vllm serve QuantTrio/Qwen3-235B-A22B-Instruct-2507-AWQ \
    --tensor-parallel-size 2 --host 127.0.0.1 --port 8001 --max-model-len 32768

CUDA_VISIBLE_DEVICES=2,3 vllm serve QuantTrio/Qwen3-235B-A22B-Instruct-2507-AWQ \
    --tensor-parallel-size 2 --host 127.0.0.1 --port 8002 --max-model-len 32768

vLLM also has --pipeline-parallel-size, which splits a model by layers; CloudRift used it for its multi-GPU RTX tests in October 2025 and tensor parallelism in November 2025. If throughput matters, run vllm bench serve against both on your own model.

Option B: four replicas behind a load balancer

When the model fits on one card, start one server per card. CUDA_VISIBLE_DEVICES decides which card a process sees; each replica gets its own port on the loopback interface. A systemd template unit keeps the four identical:

cat > /etc/vllm.env <<'EOF'
VLLM_API_KEY=paste-the-key-here
EOF
chmod 600 /etc/vllm.env

cat > /etc/systemd/system/vllm@.service <<'EOF'
[Unit]
Description=vLLM replica on GPU %i
After=network-online.target
Wants=network-online.target

[Service]
User=vllm
Group=vllm
WorkingDirectory=/home/vllm
EnvironmentFile=/etc/vllm.env
Environment=CUDA_VISIBLE_DEVICES=%i
ExecStart=/home/vllm/.venv/bin/vllm serve Qwen/Qwen3.8-27B --host 127.0.0.1 --port 800%i --max-model-len 32768
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now vllm@0 vllm@1 vllm@2 vllm@3

The instance name is the GPU index, so vllm@2 runs on GPU 2 and listens on port 8002. Start the first one alone the first time, so the 55.6 GB download happens once and the other three find it in the cache.

Check: nvidia-smi shows four processes, one per card, and each of curl http://localhost:8000/health to 8003 returns 200.

Put a load balancer in front. Caddy does TLS and balancing in one file; its reverse_proxy directive accepts several upstreams, a balancing policy (the default is random; least_conn picks the upstream with the fewest current requests) and an active health check. In /etc/caddy/Caddyfile:

llm.example.com {
	reverse_proxy :8000 :8001 :8002 :8003 {
		lb_policy least_conn
		health_uri /health
	}
}
sudo ufw allow 80
sudo ufw allow 443
sudo systemctl reload caddy

least_conn suits LLM traffic, where requests last from a second to minutes. A replica that is still loading its model fails the health check and receives no traffic. CloudRift's benchmarks used the same replica design with NGINX as the balancer.

Security: the replicas listen on 127.0.0.1 only; ports 8000 to 8003 are never opened in ufw; the only public entry is HTTPS on 443 with the API key required.

How does Ollama spread a model across several GPUs?

Ollama needs no flag for multiple GPUs. Its FAQ describes the behaviour: "If the model will entirely fit on any single GPU, Ollama will load the model on that GPU... If the model does not fit entirely on one GPU, then it will be spread across all the available GPUs." So ollama pull qwen3:235b followed by a prompt loads the 142 GB model across the cards without further configuration.

ollama pull qwen3:235b
ollama run qwen3:235b "ready"
ollama ps
nvidia-smi --query-gpu=index,memory.used,memory.total --format=csv

Check: ollama ps shows 100% GPU and the memory query shows which cards hold the model. Three settings matter on this server:

Ollama is the quick way to try a 142 GB or 290 GB model; for many concurrent users, vLLM's batching justifies the extra setup. The Ollama and Open WebUI guide shows where the environment variables go.

System RAM, downloads and storage: plan before you pull

RAM. The server has 384 GB of VRAM and 128 GB of system memory, so the largest models are bigger than RAM. Two consequences. A 243 GB or 290 GB weight file cannot sit in the page cache, so every cold start reads it from NVMe again; keep models loaded (OLLAMA_KEEP_ALIVE=-1, or a vLLM service that stays up) instead of loading on demand. And there is no useful fallback: a model that does not fit in VRAM would run partly on the CPU from system RAM, far slower, so treat 384 GB as a hard limit.

Storage. Two 1.92 TB drives give 1.92 TB usable as a RAID 1 mirror, or 3.84 TB as independent drives with no protection against a drive failure. Add up what you plan to keep:

Set of modelsDisk space
The four Ollama tags above (142 + 245 + 243 + 290 GB)920 GB
Both AWQ repositories for vLLM (124.1 + 252.3 GB)376.4 GB
One BF16 original for your own quantisation: Qwen3-235B470.2 GB
All of the above1,766.6 GB, 92% of a 1.92 TB mirror

Ollama and Hugging Face keep separate caches (/usr/share/ollama/.ollama/models and ~/.cache/huggingface; move them with OLLAMA_MODELS and HF_HOME). Public weights can be downloaded again, but a dedicated server is a single machine: fine-tuned adapters, datasets and logs you must keep need a copy off the server.

Downloads. The port is 1 Gbps with unlimited traffic, so pulling models costs nothing extra, but it takes time. At the full line rate of 125 MB per second, 290 GB takes at least 39 minutes and 470.2 GB at least 63 minutes; the real rate depends on the far end.

Monitoring each GPU

With four cards an average hides the problem, so watch them individually. From the shell:

nvidia-smi --query-gpu=index,utilization.gpu,memory.used,memory.total,temperature.gpu,power.draw --format=csv -l 5

For history and alerts, NVIDIA's DCGM exporter publishes per-GPU metrics for Prometheus on port 9400. Its documented run command, with the port bound to the loopback interface so that it is not exposed:

export DCGM_EXPORTER_TAG=latest
docker run -d --rm \
    --name dcgm-exporter \
    --gpus all \
    --cap-add SYS_ADMIN \
    -p 127.0.0.1:9400:9400 \
    nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}
curl --fail http://localhost:9400/metrics

Check: the output contains metric lines starting with DCGM_, each labelled with a GPU index. Each vLLM replica also exposes Prometheus metrics on /metrics. The Grafana dashboard guide for GPU servers turns both sources into panels.

Published multi-GPU results, and where other hardware wins

MassiveGRID does not publish benchmark figures for this server. CloudRift has published two vLLM studies on RTX PRO 6000 cards that show what to expect from each layout (checked on 16 November 2026; its cards were the Workstation Edition where stated, these servers carry the Max-Q edition):

For models that fit one card, four replicas on this server are a strong layout: CloudRift measured a single RTX PRO 6000 slightly ahead of a single H100 on such a model. For one very large model across all cards, and for large-scale training, H100 and H200 class systems are the better tool; MassiveGRID quotes those on the H100, H200 and B200 page. For reference, four of the same cards on demand for a 730-hour month were $13,139.80 at Google Cloud (g4-standard-192, Iowa), $12,097.68 at AWS (g7e.24xlarge, us-east-1) and $8,760.00 at Akamai (listed at $12.00 an hour) on 16 November 2026, so the break-even against this server is about 331, 360 and 497 hours a month. Below that many hours a month, paying by the hour costs less, and at low volume a pay-per-token API gives access to frontier models without a server. The four-card server earns its $5,966.44 when it is busy most of the day, when data has to stay on hardware that is yours alone, or when you need 384 GB at a fixed price with unlimited traffic. The GPU dedicated server plans and the multi-GPU server page compare it with the single-card servers.

What goes wrong on a multi-GPU server

SymptomCauseFix
Out of memory while loading with tensor parallelismWeights fit, but the cache for the requested context does notLower --max-model-len (32,768 is a sensible start for the largest models).
Only one card fills up, then the load fails--tensor-parallel-size missing, or CUDA_VISIBLE_DEVICES restricts the process to one cardCheck the unit file and the environment; run nvidia-smi during the load.
Ollama shows a CPU percentage in ollama psModel plus context exceeds the VRAM that is free, often because another model is still loadedLower OLLAMA_CONTEXT_LENGTH, unload other models with ollama stop, or use a smaller build.
Cold starts take many minutesHundreds of gigabytes are read from disk on every loadKeep the model loaded; do not restart services casually.
Clients get errors from the balancer while replicas loadNo health check, so traffic reaches a replica that is not readyAdd health_uri /health to the Caddy block.

Frequently Asked Questions

What does --tensor-parallel-size do in vLLM?

It splits one model across several GPUs in the same server, so that a model too large for one card can be served. The default is 1. vLLM's documentation advises using it only when the model does not fit on a single GPU but fits on one node, for example --tensor-parallel-size 4 on a node with four GPUs. For a model that fits one card, separate replicas give more throughput.

How much VRAM do I need to run a 235B model?

For Qwen3 235B it depends on the build: the default Ollama tag is a 142 GB download, an AWQ 4-bit repository for vLLM is 124.1 GB, the q8 build is 250 GB and the original BF16 weights are 470.2 GB. The 4-bit builds fit on two 96 GB cards with room for context, q8 needs three to four, and BF16 does not fit in 384 GB. Add memory for the context on top of the weights.

Can I run four different models on a four-GPU server?

Yes. Start one server process per card with CUDA_VISIBLE_DEVICES=0 to 3 and a different port each, and route by hostname or path in the reverse proxy. Each model must fit in 96 GB with its cache. You can also mix layouts, for example one model on two cards with --tensor-parallel-size 2 and two single-card models.

Does Ollama use all four GPUs automatically?

Only when it has to. Ollama's FAQ says a model that fits entirely on one GPU is loaded on that GPU, and a model that does not fit on one is spread across all available GPUs. Use CUDA_VISIBLE_DEVICES in the service environment to limit which cards it may use.

What does the 4x RTX PRO 6000 server cost per GPU?

The server is $5,966.44 per month with a one-time setup fee of the same amount, which is $1,491.61 per card per month, $2.04 per GPU-hour over a 730-hour month, and $15.54 per GB of VRAM per month. It includes a Xeon Gold 5412U, 128 GB of DDR5 ECC, 2x 1.92 TB NVMe, a 1 Gbps port with unlimited traffic and a dedicated IPv4 address. It is listed with the single-card servers on the GPU dedicated servers page.

Four RTX PRO 6000 cards, 384 GB, one tenant

The MassiveGRID 4x RTX PRO 6000 server is bare metal with four 96 GB RTX PRO 6000 Blackwell cards, a Xeon Gold 5412U, 128 GB of DDR5 ECC and 2x 1.92 TB NVMe for $5,966.44 per month ($2.04 per GPU-hour over a 730-hour month). It has a 1 Gbps port with unlimited traffic, a dedicated IPv4 address and root access, hosted in an ISO 27001 certified data center.

See Multi-GPU Servers

Further Reading