A Dedicated AI Inference Server for Production APIs
An AI inference server keeps a model loaded on a GPU and answers API requests from your applications, all day. MassiveGRID rents bare-metal servers for that job: one RTX PRO 6000 Blackwell with 96 GB of vRAM for $1,656.15 a month ($2.27 an hour equivalent), four of them for $5,966.44, or a 24 GB entry server for $449.99, with no charge per token or per hour. The machines are listed on the GPU dedicated servers page; this page covers what decides throughput and what it costs per token.
What is an inference server, and what determines its throughput?
Inference server: a machine that loads model weights into GPU memory once and then serves requests over an API, as opposed to a training cluster or a notebook. For language models the usual software is vLLM, which exposes OpenAI-compatible endpoints. Four things decide how many tokens per second one card delivers:
- Batching. A GPU serving one request at a time is mostly idle. vLLM uses continuous batching of incoming requests, so total throughput rises steeply with concurrency while each user's stream slows only a little. Akamai, a cloud vendor that sells this GPU, published figures for a 49B model on one RTX PRO 6000 Blackwell Server Edition (NVIDIA NIM, FP8, 30 October 2025): 27 tokens per second at one concurrent request, 2,256 at 100 and 3,606 at 200.
- Quantisation. Llama 3.3 70B is 141 GB at 16-bit, 75 GB at 8-bit and 43 GB at 4-bit. Fewer bits mean a smaller model, and more memory left for batching, at some cost in output quality that you should test on your own prompts.
- Context length. Every concurrent request holds its context in GPU memory. Long documents cut the number of requests that fit in a batch: in StorageReview's tests the same card produced 2,283 output tokens per second on short chats and 608 on 8,192-token documents.
- Model size and card. Larger models are slower per token, and memory bandwidth differs by card: 1,792 GB/s on the RTX PRO 6000 Blackwell workstation cards, 432 GB/s on the RTX PRO 4000 SFF.
Which inference server for which workload?
| Workload | Server | vRAM | Monthly | Per hour equivalent |
|---|---|---|---|---|
| One 14B to 27B model at 4-bit for internal tools, embeddings, Whisper | RTX PRO 4000 | 24 GB | $449.99 | $0.62 |
| A 70B or 120B model, or a 27B to 32B model at 16-bit with many concurrent users | RTX PRO 6000 | 96 GB | $1,656.15 | $2.27 |
| Up to four isolated services on one card with MIG | RTX PRO 6000 | 4x 24 GB | $1,656.15 | $2.27 |
| The same card with ECC system memory, 48 threads for tokenising, retrieval and preprocessing, up to 768 GB of RAM | RTX PRO 6000 Xeon | 96 GB | $2,485.05 | $3.40 |
| Four replicas for throughput, or a model that needs more than 96 GB | 4x RTX PRO 6000 | 384 GB | $5,966.44 | $8.17 |
On the four-card server follow vLLM's own guidance: if the model fits on one GPU, do not distribute it; run one replica per card behind a load balancer. Only a model that does not fit on one card should be split with --tensor-parallel-size 4. The multi-GPU server page covers both layouts, and LLM hosting lists which models fit in 24, 96 and 384 GB.
vLLM on a dedicated GPU: the facts that matter
From the vLLM documentation, read on 16 November 2026:
- Version and requirements: 0.33.0, released 2 November 2026, Python 3.11 to 3.14, wheels built against CUDA 12.9.
- API: OpenAI-compatible
/v1/chat/completions,/v1/completions,/v1/responses,/v1/embeddingsand/v1/audio/transcriptionson port 8000, plus rerank and score endpoints,/healthand a Prometheus/metricsendpoint. - Authentication: an API key with
--api-keyor theVLLM_API_KEYvariable. That is a shared key, not user management, so keep a TLS reverse proxy in front. - Memory:
--gpu-memory-utilizationsets the fraction of GPU memory vLLM takes, and--max-model-lencaps the context. Lowering the context cap is the first fix when a model does not load.
Multi-Instance GPU. NVIDIA lists the RTX PRO 6000 Blackwell for up to four MIG instances, each with "its own high-bandwidth memory, cache, and compute cores": four of 24 GB, two of 48 GB, or one of 96 GB. That turns one card into four isolated inference endpoints, for example a chat model, an embedding model, a reranker and Whisper, none of which can take memory from the others. The caveats are real. On the Max-Q edition in these servers MIG needs driver 575.51.03 or later and vBIOS 98.02.6A.00.00 or later, the card ships in graphics mode and must be switched to compute mode with NVIDIA's DisplayModeSelector tool first, and MIG mode does not persist across a reboot. If MIG is a requirement, confirm with support before ordering that the card in your server meets those conditions. The RTX PRO 4000 does not support MIG. Commands are in splitting an RTX PRO 6000 with MIG.
Published benchmarks, and flat monthly cost vs per-token and per-hour pricing
MassiveGRID publishes no throughput figures of its own. These are third-party results for the RTX PRO 6000 Blackwell on vLLM, all on the 600 W Workstation Edition; the servers here use the 300 W Max-Q edition, for which we have found no published figure, so expect lower numbers and measure your own workload.
| Source | Test | RTX PRO 6000 Blackwell | For comparison |
|---|---|---|---|
| StorageReview, updated 3 November 2026 | Ministral 3 14B FP8, 32 streams, 512 tokens in and out | 2,283 output tokens/s | L40S 995; H100 NVL 3,096 |
| CloudRift, 9 October 2025 | Llama 3.3 70B AWQ INT4, 400 concurrent, 1,000 tokens in and out | 1,031 tokens/s | Two RTX 5090: 1,230 |
| CloudRift, 27 November 2025 | GLM-4.5-Air AWQ 4-bit, 256 to 512 concurrent | 3,140 tokens/s; eight cards as eight replicas 15,713 | One H100: 2,987 |
Cost per token. A flat price means the cost per token depends on how busy the card is. Take CloudRift's 1,031 tokens per second for Llama 3.3 70B: over 730 hours of constant saturation that is about 2.7 billion output tokens, or $0.61 per million on the $1,656.15 server, with nothing extra for input tokens. At 25% utilisation it is $2.44 and at 5% it is $12.22. For scale, the list prices of premium API models on 16 November 2026 were $2 per million input and $10 per million output tokens for OpenAI gpt-6.1-sol and Claude Sonnet 5.5, $2 and $12 for Gemini 3.1 Pro (prompts up to 200,000 tokens), $4 and $20 for Claude Opus 5.5, $8 and $40 for Claude Fable 5.2 and $10 and $50 for OpenAI gpt-6-astra. At CloudRift's test mix of 1,000 tokens in and 1,000 out, a $2 and $10 model costs $12 for each million output tokens, which the server matches at about 5% utilisation; below that, paying per token costs less. Those are proprietary models and Llama 3.3 70B is an open-weight one, and the benchmark card is the 600 W edition, so compare output quality and measure your own throughput before you compare prices.
Per hour. AWS g7e.2xlarge with the same class of card is $3.36312 an hour on demand in N. Virginia, so the dedicated server costs less above about 492 hours a month. Against Google Cloud's g4-standard-48 at $4.49993 in Iowa or Oracle Cloud's list price of $4.50 the point is about 368 hours, against Azure's NC144lds at $5.50 in East US about 301 hours, and against RunPod Secure Cloud at $2.69 about 616 hours (all checked on 16 November 2026, before storage and egress).
Where the alternatives win: per-token APIs for spiky or low traffic and for the quality of frontier models; hourly cloud for batch jobs that end; H100-class cards for the largest models, where CloudRift measured about three times the throughput on an FP8 model across eight GPUs. Where a dedicated inference server wins: your own fine-tuned or private models, data that must stay on a machine only you use, steady traffic for most of the month, no preemption or stock-outs, unlimited traffic on the port and a bill that is the same every month. More in GPU inference cost optimisation and on the GPU cloud alternative page.
Four dedicated inference servers
Bare metal for one customer with root access, a 1 Gbps port with unlimited traffic and one dedicated IPv4, hosted in an ISO 27001 certified data center. A 10 Gbps uplink is an option on the Xeon server.
NVIDIA RTX PRO 6000
- 3,511 AI TOPS
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
NVIDIA RTX PRO 6000 Xeon
- 3,511 AI TOPS
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
4x NVIDIA RTX PRO 6000
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
NVIDIA RTX PRO 4000
- 770 AI TOPS
- CUDA and FP4 support
- 1 dedicated IPv4 included
Every RTX PRO server includes: a 1 Gbps port with unlimited traffic, one dedicated IPv4 address, hosting in an ISO 27001 certified data center, and your choice of Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS Stream, openSUSE or Arch Linux, or a rescue system to install your own OS. Billing runs monthly, quarterly, semi-annually, annually, biennially or triennially, and the setup fee is charged once with the first invoice.
Options on the RTX PRO 6000 Xeon: 512 GB DDR5 ECC with 2x 1.92 TB NVMe for $994.68/mo more, 768 GB DDR5 ECC with 4x 3.84 TB NVMe for $1,989.37/mo more, and a 10 Gbps uplink with 20 TB of traffic included for $71.29/mo more.
Deploy vLLM on the inference server
The server is delivered with a clean operating system (Ubuntu 24.04 LTS by default) and root access. Nothing is pre-installed.
- Install the NVIDIA driver and reboot.
nvidia-smimust list the card before you go on:sudo apt install linux-headers-$(uname -r) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt update sudo apt install nvidia-open sudo reboot - Install vLLM in a virtual environment, as its documentation describes:
uv venv --python 3.12 --seed --managed-python source .venv/bin/activate uv pip install vllm --torch-backend=auto - Serve a model with an API key and check that it answers:
Run theexport VLLM_API_KEY="$(openssl rand -hex 32)" vllm serve Qwen/Qwen3-32B --max-model-len 32768 curl http://localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"curlcheck from a second terminal. For four cards add--tensor-parallel-size 4only if the model does not fit on one. - Put TLS in front. Keep port 8000 closed in the firewall and publish the API through a reverse proxy with a certificate.
- Measure. Replay your real prompts at the concurrency you expect and record output tokens per second and time to first token. Change one setting at a time: quantisation, context cap, then concurrency.
A dedicated server is one machine. Run a second server behind a load balancer if the API must survive a hardware fault, and keep weights and configuration backed up off the server. The full walkthrough is a self-hosted LLM API with vLLM.
Monitor the GPU with DCGM and Prometheus
Two sources cover an inference server. vLLM's /metrics endpoint reports the serving side in Prometheus format. NVIDIA's DCGM exporter (4.9.0 at the time of writing) reports the hardware: utilisation, memory, temperature and power, as metrics prefixed DCGM_ on port 9400. With Docker and the NVIDIA Container Toolkit installed, NVIDIA documents this command:
export DCGM_EXPORTER_TAG=latest
docker run -d --rm \
--name dcgm-exporter \
--gpus all \
--cap-add SYS_ADMIN \
-p 9400:9400 \
nvcr.io/nvidia/k8s/dcgm-exporter:${DCGM_EXPORTER_TAG}
curl --fail http://localhost:9400/metricsScrape both endpoints with Prometheus and alert on GPU memory near the limit and on a growing request queue: those two signals tell you when to cut context, add a replica or move up a server. Do not publish port 9400 to the internet. Dashboards are covered in GPU monitoring with Grafana.
Related inference pages
Hardware, models and simpler stacks.
RTX PRO 6000 Server
The card and the three configurations in detail.
384 GBMulti-GPU Server
Replicas or tensor parallelism on four cards.
LLMLLM Hosting
Which model, which licence, which server.
OllamaOllama Hosting
The simpler stack for one team and a chat interface.
QuoteNVIDIA H100 Server
H100, H200 and B200 systems for the largest models.
CostGPU Cloud Alternative
Dedicated against hourly and serverless GPU pricing.
Inference guides
Self-Host an LLM API with vLLM on a GPU Server
Serve an OpenAI-compatible LLM API with vLLM: continuous batching, PagedAttention, the flags that matter, systemd setup, nginx and throughput benchmarking.
GuidevLLM on One 96 GB GPU: Serving Llama 3.3 70B, Qwen, gpt-oss-120b and DeepSeek Distills
vLLM serve Llama 3.3 70B on one 96 GB GPU: the 141.1 GB original does not fit, a 39.8 GB AWQ build does. Flags, API key, curl test, systemd unit.
GuideNVIDIA MIG on the RTX PRO 6000: Split One 96 GB GPU into Isolated Instances
NVIDIA MIG on the RTX PRO 6000 Blackwell splits 96 GB into four 24 GB or two 48 GB instances. Prerequisites, nvidia-smi commands and Docker device selection.
GuidevLLM Multi-GPU on Four RTX PRO 6000 Cards: Tensor Parallelism, Replicas and 235B Models
vLLM multi GPU on four 96 GB cards: --tensor-parallel-size 4 for one model up to 384 GB, or four replicas behind a load balancer. What a 235B model needs.
GuideMulti-Tenant GPU Serving: Batching, Caching and Cost Per Token
Cut GPU inference cost: why batching is nearly free, the KV cache as your capacity limit, prefix caching, prefill vs decode, MIG and adapters, cost per million tokens.
GuideMonitoring GPU Utilization: nvidia-smi, DCGM and Grafana
Monitor GPUs properly: the metrics that distinguish busy from starved from throttling, DCGM exporter with Prometheus, four alerts worth having, and reading the graphs.
Frequently Asked Questions
Put your model on a dedicated inference server
One RTX PRO 6000 Blackwell with 96 GB for $1,656.15 a month, four for $5,966.44. Same bill every month, whatever the token count.