To run a 70B model on one GPU with vLLM you need a quantised build: meta-llama/Llama-3.3-70B-Instruct is 141.1 GB at full precision and does not fit in 96 GB, while the AWQ INT4 build is 39.8 GB and does. Models of 27B to 32B (Qwen3.8-27B at 55.6 GB, Gemma 4 31B at 62.5 GB, Qwen3-32B at 65.5 GB) fit at full precision, and OpenAI states that gpt-oss-120b fits a single 80 GB GPU. The command is vllm serve <repository> with --api-key, --max-model-len and --gpu-memory-utilization, and the result is an OpenAI-compatible API on port 8000.

This guide uses vLLM 0.33.0 on a MassiveGRID RTX PRO 6000 server with a clean Ubuntu 24.04 install: what fits, the install, the Hugging Face token for gated models, the serve flags, a curl test, a systemd unit, and what published benchmarks say about throughput.

Which models fit on one 96 GB GPU?

vLLM loads models in Hugging Face format, and the first number to look at is the size of the weight files. The weights must fit in the 96 GB of the RTX PRO 6000 with room left for the key-value cache, which is what holds every conversation in progress. More spare memory means more concurrent requests or longer contexts. The sizes below are the safetensors files in each repository as the Hugging Face API reported them on 16 November 2026.

Model (Hugging Face repository)WeightsPrecisionLicenceOn one 96 GB card
Qwen/Qwen3.8-27B55.6 GBBF16 (full)Apache 2.0Fits, about 40 GB left for cache
google/gemma-4-31b-it62.5 GBBF16 (full)Apache 2.0Fits, about 33 GB left
Qwen/Qwen3-32B65.5 GBBF16 (full)Apache 2.0Fits, about 30 GB left
deepseek-ai/DeepSeek-R1-Distill-Qwen-32B65.5 GBBF16 (full)MITFits, about 30 GB left
mistralai/Mistral-Small-3.2-24B-Instruct-2506about 55 GB of GPU RAM in bf16 or fp16, per its model cardBF16 (full)Apache 2.0Fits
openai/gpt-oss-120bMXFP4-quantised by OpenAI; the card says it fits a single 80 GB GPUMXFP4 + BF16Apache 2.0Fits
meta-llama/Llama-3.3-70B-Instruct141.1 GBBF16 (full)Llama 3.3 Community License, gatedDoes not fit
ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT439.8 GBAWQ 4-bitllama3.3Fits, about 56 GB left
deepseek-ai/DeepSeek-R1-Distill-Llama-70B141.1 GBBF16 (full)MITDoes not fit; needs a quantised build
deepseek-ai/DeepSeek-R1 (the 671B model)688.6 GBFP8 + BF16MITDoes not fit on this server or on four cards

Three conclusions. Models in the 27B to 32B class run at full precision on one card, with no quantisation trade-off. A 70B model needs a 4-bit build: 141.1 GB of BF16 weights is larger than 96 GB, while the AWQ INT4 build of Llama 3.3 70B is 39.8 GB. And gpt-oss-120b, the largest model in the table that fits, was published by OpenAI already quantised for a single GPU of this class. "DeepSeek on one GPU" means one of the R1 distills (the 32B one is a Qwen-based model, the 70B one a Llama-based model), not R1 itself. For the reasoning behind these numbers, see how much VRAM a model needs.

The same rules apply on the 24 GB RTX PRO 4000 at a smaller scale: a 14B model is already 29 GB to 30 GB in 16-bit precision, more than the card holds, so that card is for quantised builds and smaller models. This guide is written for the 96 GB servers in the GPU dedicated server range.

Step 1: install vLLM 0.33.0

You need the NVIDIA driver working first (nvidia-smi must print the card); the first-hour guide for a new GPU server covers that. vLLM 0.33.0 was released on 2 November 2026 and requires Python 3.11 to 3.14. Its documentation recommends installing with uv into a virtual environment, so nothing touches the system Python.

Create a dedicated user and install uv with Astral's documented script:

useradd -m -s /bin/bash vllm
su - vllm
curl -LsSf https://astral.sh/uv/install.sh | sh
source ~/.profile

The installer puts uv in ~/.local/bin; sourcing ~/.profile (or logging in again) adds that directory to the path on Ubuntu. Then the four commands from vLLM's GPU installation page:

uv venv --python 3.12 --seed --managed-python
source .venv/bin/activate
uv pip install vllm --torch-backend=auto
vllm download-kernels

--torch-backend=auto selects the PyTorch build that matches the installed driver; vLLM's binaries are built against CUDA 12.9, and you do not need the CUDA Toolkit on the host. To hold the version, install vllm==0.33.0 instead.

Check: uv pip show vllm prints version 0.33.0 and python -c "import torch; print(torch.cuda.is_available())" prints True.

If you prefer containers, the documented image is vllm/vllm-openai, run with --runtime nvidia --gpus all, a volume for ~/.cache/huggingface and --ipc=host (vLLM needs the host's shared memory). Publish its port as -p 127.0.0.1:8000:8000, not -p 8000:8000: Docker's published ports bypass ufw.

Step 2: Hugging Face token for gated models

vLLM downloads the model from Hugging Face on first start. Qwen, gpt-oss, Gemma 4 and the DeepSeek distills are open repositories. Meta's Llama repositories and Google's Gemma 3 are gated: you accept the licence on the model page while logged in, create a read token in your Hugging Face account settings, and give it to vLLM through the HF_TOKEN environment variable.

export HF_TOKEN=hf_your_token_here

Without it, a gated download fails with a 401 error. The community AWQ build of Llama 3.3 70B used below is not gated, but it carries the same Llama 3.3 licence, so read it before commercial use. Downloads go to ~/.cache/huggingface; a 55 GB to 65 GB model takes that much disk, and the gpt-oss-120b repository holds 195.8 GB in total because it includes other formats, so check free space with df -h first.

Step 3: vllm serve with the flags that matter

Start with a model that fits at full precision. This command serves Qwen3.8-27B on the loopback interface with an API key:

export VLLM_API_KEY=$(openssl rand -hex 32)
echo $VLLM_API_KEY

vllm serve Qwen/Qwen3.8-27B \
    --host 127.0.0.1 \
    --port 8000 \
    --max-model-len 32768 \
    --gpu-memory-utilization 0.92

The first start downloads 55.6 GB and then loads it; watch nvidia-smi in a second terminal. The server is ready when the log shows the API routes and the port.

FlagWhat it doesGuidance on a 96 GB card
--api-key or VLLM_API_KEYThe server checks the key in the request header. Several keys can be passed for rotation.Always set one. The environment variable keeps it out of the process list.
--gpu-memory-utilization"The fraction of GPU memory to be used for the model executor, which can range from 0 to 1."The documented default is 1.0 with the extensible KV cache and 0.92 otherwise. Lower it only if something else must share the card.
--max-model-lenMaximum context per request; derived from the model's configuration if unset.Qwen3.8-27B declares 262,144 tokens. Set what you need (8,192 to 32,768 for chat) so the cache serves more requests.
--host and --portBind address and port; the server documentation uses port 8000.Bind to 127.0.0.1 and put a TLS reverse proxy in front.
--served-model-nameThe model name clients send.Useful to hide a long repository path behind a stable name.
--tensor-parallel-sizeSplits one model over several GPUs; default 1.Leave at 1. vLLM's guidance: if the model fits on a single GPU, do not distribute it.
--quantizationFalls back to the quantisation in the model's own configuration.Rarely needed: AWQ repositories declare it themselves.

If loading stops with a CUDA out-of-memory error, the weights fit but the requested context does not: lower --max-model-len first.

Step 4: test the OpenAI-compatible API with curl

vLLM serves the OpenAI API shape: /v1/models, /v1/chat/completions, /v1/completions, /v1/responses and /v1/embeddings, plus /health and Prometheus metrics on /metrics. From the server itself:

curl http://localhost:8000/v1/models \
    -H "Authorization: Bearer $VLLM_API_KEY"

curl http://localhost:8000/v1/chat/completions \
    -H "Content-Type: application/json" \
    -H "Authorization: Bearer $VLLM_API_KEY" \
    -d '{
        "model": "Qwen/Qwen3.8-27B",
        "messages": [
            {"role": "system", "content": "You are a helpful assistant."},
            {"role": "user", "content": "Explain tensor parallelism in two sentences."}
        ]
    }' 

Check: the first call lists the model, the second returns a JSON completion, and the same call without the header is rejected. Any client built for the OpenAI API works by changing two settings: the base URL (https://llm.example.com/v1 once the proxy is up) and the key. That includes the OpenAI Python package, Open WebUI and the tools described in building a self-hosted RAG stack.

Step 5: Llama 3.3 70B and gpt-oss-120b on one card

Llama 3.3 70B. The original repository is 141.1 GB in BF16, which is 45 GB more than the card has. Serve the 4-bit AWQ build, 39.8 GB, which is the repository CloudRift used in the benchmark quoted below:

vllm serve ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4 \
    --host 127.0.0.1 \
    --port 8000 \
    --served-model-name llama-3.3-70b \
    --max-model-len 8192

With about 56 GB free after the weights, this model has more cache than the full-precision 32B models, so it handles many concurrent requests or a longer --max-model-len. The trade-off is quantisation: 4-bit weights lose some accuracy against the BF16 original. Compare both on your own prompts before deciding; a full-precision 27B or 32B model is sometimes the better choice than a 4-bit 70B.

gpt-oss-120b. OpenAI published this model already quantised (MXFP4 on the mixture-of-experts weights, 117B parameters with 5.1B active) and its card states that it fits a single 80 GB GPU. The card's vLLM command is simply:

vllm serve openai/gpt-oss-120b

Add the same --host, --port and key settings as above. Check after either start: nvidia-smi shows one process holding most of the 96 GB (vLLM allocates its cache when it starts, so high memory use is normal), and the curl test from step 4 answers with the new model name.

Step 6: run it as a systemd service behind TLS

A server started in a terminal dies with the session. Put the key and token in an environment file readable only by root, and let systemd keep the process running:

cat > /etc/vllm.env <<'EOF'
VLLM_API_KEY=paste-the-key-here
HF_TOKEN=hf_your_token_here
EOF
chmod 600 /etc/vllm.env

cat > /etc/systemd/system/vllm.service <<'EOF'
[Unit]
Description=vLLM OpenAI-compatible server
After=network-online.target
Wants=network-online.target

[Service]
User=vllm
Group=vllm
WorkingDirectory=/home/vllm
EnvironmentFile=/etc/vllm.env
ExecStart=/home/vllm/.venv/bin/vllm serve Qwen/Qwen3.8-27B --host 127.0.0.1 --port 8000 --max-model-len 32768
Restart=always
RestartSec=10

[Install]
WantedBy=multi-user.target
EOF

systemctl daemon-reload
systemctl enable --now vllm
journalctl -u vllm -f

Check: curl http://localhost:8000/health returns 200 once the model is loaded, and again after systemctl restart vllm.

Security. The unit binds to 127.0.0.1, so the API is not reachable from outside yet. To publish it, install a reverse proxy with a certificate (Caddy needs two lines: the hostname and reverse_proxy :8000), open only ports 80 and 443 in ufw, and keep the API key mandatory. Do not open port 8000. The /metrics endpoint is for your monitoring, not for the public; the Prometheus and Grafana guide for GPU servers shows how to scrape it locally.

What throughput to expect: published figures

MassiveGRID does not publish benchmark numbers for these servers. Two independent sources have measured the RTX PRO 6000 Blackwell with vLLM; StorageReview and the second CloudRift test name the 600 W Workstation Edition, while these servers carry the 300 W Max-Q edition of the same 96 GB card, so treat the figures as an upper reference, not a promise. All figures were checked on 16 November 2026.

Source and dateSetupModelOutput tokens per second, one RTX PRO 6000
CloudRift, 9 October 2025vLLM, 400 concurrent requests, 1,000 tokens in and out, --max-model-len 8192, --kv-cache-dtype fp8Llama 3.3 70B AWQ INT41,031
CloudRift, 9 October 2025SameQwen3-Coder-30B-A3B AWQ 4-bit8,425
CloudRift, 27 November 2025vLLM, 256 to 512 concurrent requests, 1,000 tokens in and outGLM-4.5-Air AWQ 4-bit3,140 (H100 in the same test: 2,987)
StorageReview, updated 3 November 2026vLLM 0.27.1, 32 concurrent streams, 512 tokens in and outMinistral 3 14B FP82,283 (H100 NVL: 3,096; L40S: 995)
StorageReview, updated 3 November 2026Same, 8,192 tokens in and 1,024 outMinistral 3 14B FP8608

These are totals across all concurrent requests, not the speed one user sees. They show the pattern that matters for planning: throughput falls as the model grows (8,425 for a 30B mixture-of-experts model against 1,031 for a dense 70B in the same test) and as prompts get longer (2,283 against 608).

To measure your own workload, vLLM ships a load generator. With the server running:

vllm bench serve \
    --model Qwen/Qwen3.8-27B \
    --base-url http://localhost:8000 \
    --endpoint /v1/chat/completions \
    --backend openai-chat \
    --dataset-name random \
    --random-input-len 1000 \
    --random-output-len 1000 \
    --max-concurrency 64 \
    --num-prompts 256

It reports total throughput, time to first token and latency percentiles. Run it at the concurrency you expect, then at twice that. The options are the ones CloudRift published with its results. If your server requires a key, vllm bench serve --help lists how to pass it; for a first measurement it is simpler to benchmark before you enable the key.

What goes wrong, and when one GPU is the wrong tool

SymptomCauseFix
CUDA out of memory while loadingWeights plus the cache for the requested context exceed 96 GBLower --max-model-len; if the weights alone come close to 96 GB, use a quantised build.
401 or "gated repo" during downloadNo token, or the licence was not accepted on the model pageAccept the licence, export HF_TOKEN, restart.
Download stops with no space leftModel cache on a small root filesystemCheck df -h; point HF_HOME at the larger volume.
Requests wait while GPU use is lowClient sends requests one at a timevLLM batches concurrent requests; send them in parallel.
nvidia-smi shows most of the 96 GB in use with no trafficvLLM allocates its cache up to the configured memory fraction when it startsExpected. Lower --gpu-memory-utilization only to share the card.
The API answers from the internet without a keyStarted without a key and bound to all interfaces, or a Docker port was publishedBind to 127.0.0.1, set the key, use the TLS proxy.

One 96 GB card at $1,656.15 per month ($2.27 per hour over a 730-hour month) is not always the right answer. A pay-per-token API costs less at low volume and gives access to frontier models that no single card runs. Hourly cloud GPUs cost less for short or bursty jobs: the server is billed for the whole month whether it is busy or idle. H100 and H200 class systems are the better tool for large-scale training and for models that need several cards working as one; MassiveGRID quotes those through sales. A dedicated 96 GB server wins when the model runs most of the day, when data must stay on hardware that is yours alone, and when you want a fixed bill with unlimited traffic. The AI inference server page sizes the four servers by workload, and the multi-GPU vLLM guide covers models that need more than 96 GB.

Frequently Asked Questions

Can vLLM serve Llama 3.3 70B on a single GPU?

Yes, on a 96 GB card with a quantised build. The full-precision repository is 141.1 GB, so it does not fit, but the AWQ INT4 build (ibnzterrell/Meta-Llama-3.3-70B-Instruct-AWQ-INT4) is 39.8 GB and leaves about 56 GB for the cache. CloudRift measured 1,031 output tokens per second for that build on one RTX PRO 6000 at 400 concurrent requests (published 9 October 2025).

What does --gpu-memory-utilization do in vLLM?

It sets the fraction of GPU memory, from 0 to 1, that vLLM may use for the model executor: the weights plus the key-value cache. In the current documentation the default is 1.0 with the extensible KV cache and 0.92 otherwise. Whatever the weights do not use becomes cache, so a high value means more concurrent requests; lower it only when another process has to share the card.

How do I protect a vLLM server with an API key?

Pass --api-key or set the VLLM_API_KEY environment variable; the server then checks the key in the request header and rejects requests without it. A key is not transport security, so also bind vLLM to 127.0.0.1 and publish it only through a reverse proxy with TLS.

Which DeepSeek model runs on one 96 GB GPU?

The R1 distills. DeepSeek-R1-Distill-Qwen-32B is 65.5 GB at full precision and fits on one card. The 70B distill is 141.1 GB and needs a quantised build, and DeepSeek-R1 itself is 688.6 GB, which is beyond a single card and beyond the 384 GB of the four-card server.

Which MassiveGRID server is this guide for?

The RTX PRO 6000 server (96 GB, Core Ultra 7 265, 128 GB of RAM, $1,656.15 per month) or the Xeon variant with 256 GB of RAM at $2,485.05 per month. Both are listed on the GPU dedicated servers page. They arrive with a clean OS and root access; drivers and vLLM are installed by you, as described here.

A 96 GB inference server that is yours alone

The MassiveGRID RTX PRO 6000 server is bare metal with one 96 GB RTX PRO 6000 Blackwell, 128 GB of DDR5 and 2x 1.92 TB NVMe for $1,656.15 per month, or with a Xeon Gold 5412U and 256 GB of ECC memory for $2,485.05 per month. Each has a 1 Gbps port with unlimited traffic, a dedicated IPv4 address and root access, hosted in an ISO 27001 certified data center.

See AI Inference Servers

Further Reading