The best GPU for LLM inference is the smallest card that holds your model and its context entirely in GPU memory. In November 2026 that means 24 GB for open models up to about 32 billion parameters at 4-bit quantization (18 to 20 GB), 96 GB for 70B-class models (43 GB) and 120B mixture-of-experts models (65 to 87 GB), and 384 GB across four cards for 235B to 480B models (142 to 290 GB). Speed comes second and is mostly a matter of memory bandwidth.
This guide ranks GPUs by use case rather than naming one winner, then gives the memory each current open model needs, with sizes read from the Ollama library and Hugging Face on 16 November 2026. For the underlying arithmetic of parameters, bits and context, see our LLM VRAM requirements guide. MassiveGRID rents three of the configurations discussed as GPU dedicated servers; the others are included because they are sometimes the better answer.
What is the best GPU for LLM inference in 2026?
It depends on what you run and how often. Six cases cover most buyers.
| Use case | Best fit | GPU memory | Example on-demand price, checked on 16 November 2026 |
|---|---|---|---|
| Private assistant for a small team, models up to 32B | RTX PRO 4000 Blackwell, or a cloud L4 or A10G | 24 GB | $449.99 a month as a MassiveGRID server; AWS g6.xlarge (L4) $0.8048 an hour |
| 70B-class and 120B models, long context, several models loaded | RTX PRO 6000 Blackwell | 96 GB | $1,656.15 a month as a MassiveGRID server; RunPod Secure Cloud $2.69, AWS g7e.2xlarge $3.36312, Google Cloud g4-standard-48 $4.49993 and Azure NC144lds $5.50 an hour |
| Lowest latency per request, largest scale | H100 or H200 | 80 GB or 141 GB | H100: Lambda $3.79 (PCIe), AWS p5.4xlarge $6.88, Azure NC40ads_H100_v5 $6.98 and Oracle $10.00 an hour. H200: Oracle $10.00 an hour |
| Models of 100 to 300 GB | 4x RTX PRO 6000, or H200 and B200 nodes | 384 GB | $5,966.44 a month as a MassiveGRID server; AWS g7e.24xlarge $16.57216 an hour |
| A developer's own desk | GeForce RTX 5090 | 32 GB | A workstation purchase; not on the large-cloud price lists we read |
| Occasional use, or under about 100 million tokens a month | No GPU: an API or an hourly instance | - | Premium API list prices from $2.00 per million input tokens (gpt-6.1-sol, Claude Sonnet 5.5) |
Three notes on the ranking. The 48 GB class (L40S, RTX 6000 Ada) is a cloud middle ground, but it holds a 43 GB model with almost no room for context, and the L40S has no MIG. The RTX 5090 is fast, yet NVIDIA's GeForce driver licence states "No Datacenter Deployment", so it is a workstation card. And the last row is serious: if your volume is below the break-even against API list prices, paying per token costs less.
The H100 class is the better choice for large-scale training and for very large models spread over many cards. H100, H200 and B200 systems are quoted by MassiveGRID sales rather than sold in the store.
How much VRAM do current open models need?
The table lists the default download size of each model on Ollama, which is 4-bit quantization (Q4_K_M) unless noted. The model needs that much GPU memory plus working memory for the context. "Yes" means the download leaves at least 10% of the memory free, "tight" means it loads with less than that, so only with a short context. The 384 GB column assumes the model is split across four 96 GB cards.
| Model (Ollama tag) | Default size | 24 GB | 96 GB | 384 GB | Licence |
|---|---|---|---|---|---|
| llama3.1:8b | 4.9 GB | yes | yes | yes | Llama 3.1 Community License |
| gemma4:12b | 8.0 GB | yes | yes | yes | Apache 2.0 |
| phi4 (14B) | 9.1 GB | yes | yes | yes | MIT |
| qwen3:14b | 9.3 GB | yes | yes | yes | Apache 2.0 |
| gpt-oss:20b | 14 GB (MXFP4) | yes | yes | yes | Apache 2.0 |
| mistral-small3.2 (24B) | 15 GB | yes | yes | yes | Apache 2.0 |
| devstral-small-2 (24B) | 15 GB | yes | yes | yes | Apache 2.0 |
| gemma3:27b | 17 GB | yes | yes | yes | Gemma Terms of Use |
| qwen3.8:27b | 18 GB | yes | yes | yes | Apache 2.0 |
| qwen3-coder:30b | 19 GB | yes | yes | yes | Apache 2.0 |
| gemma4:31b | 20 GB | yes | yes | yes | Apache 2.0 |
| qwen3:32b | 20 GB | yes | yes | yes | Apache 2.0 |
| deepseek-r1:32b (a Qwen distill) | 20 GB | yes | yes | yes | MIT |
| qwen3.6:35b | 23 GB | tight | yes | yes | Apache 2.0 |
| llama3.3:70b | 43 GB | no | yes | yes | Llama 3.3 Community License |
| deepseek-r1:70b (a Llama distill) | 43 GB | no | yes | yes | MIT |
| command-r-plus (104B) | 59 GB (Q4_0) | no | yes | yes | CC BY-NC 4.0, non-commercial |
| gpt-oss:120b | 65 GB (MXFP4) | no | yes | yes | Apache 2.0 |
| llama4:scout | 67 GB | no | yes | yes | Llama 4 Community License |
| mistral-large (123B, 2411) | 73 GB | no | yes | yes | Mistral Research License, non-commercial |
| devstral-2 (123B) | 75 GB | no | yes | yes | Modified MIT, restricted above $20M monthly revenue |
| mistral-medium-3.5 (128B) | 80 GB | no | yes | yes | Modified MIT, exceptions for large-revenue companies |
| qwen3.5:122b | 81 GB | no | yes | yes | Apache 2.0 |
| nemotron-3-super (120B) | 87 GB | no | tight | yes | NVIDIA model licence |
| qwen3:235b | 142 GB | no | no | yes | Apache 2.0 |
| llama3.1:405b | 243 GB | no | no | yes | Llama 3.1 Community License |
| llama4:maverick | 245 GB | no | no | yes | Llama 4 Community License |
| qwen3-coder:480b | 290 GB | no | no | yes | Apache 2.0 |
| deepseek-r1:671b, deepseek-v3.1:671b | 404 GB | no | no | no | MIT |
Higher precision costs memory in proportion. qwen3.8:27b is 18 GB at 4-bit, 30 GB at 8-bit and 56 GB at full 16-bit precision; llama3.3:70b is 43, 75 and 141 GB. On a 96 GB card you can therefore run a 27B to 32B model unquantized or a 70B model at 8-bit, which a 24 GB card cannot do. Check the licence column before building a product: four of the larger models above are non-commercial or revenue-restricted.
Embedding models are small (nomic-embed-text 274 MB, bge-m3 1.2 GB) and sit beside a chat model on any of these cards.
Which flagship models do not fit even 384 GB?
The newest open-weight flagships of 2026 are far larger than any single server discussed here. Their weights on Hugging Face:
- Mistral Large 4 (1.05 trillion parameters, 52 billion active, published 30 September 2026): 1,060.6 GB.
- DeepSeek V4-Pro (1.6 trillion parameters, 49 billion active): 864.7 GB.
- Qwen3.8-2.4T-A95B (2.4 trillion parameters, published 8 August 2026): 4,892 GB.
- Mistral Large 3 (675 billion parameters): 681.5 GB. Its model card names a single node of B200s or H200s for FP8.
On Ollama these appear only as cloud tags, with no local download. Running them yourself means a multi-node H200 or B200 deployment, which is a quote-level project, or using the vendor's API. For most self-hosting the practical 2026 choices are the 27B to 35B generation (Qwen3.8, Gemma 4), the 70B class and the 120B mixture-of-experts models, all of which fit one 96 GB card.
How does context length change the memory you need?
The download size is the floor. Every token of context adds to a key-value cache that also lives in GPU memory, so a model that loads comfortably with a 4,000-token context can run out of memory at 128,000. The vendors' own statements show the scale:
- The Qwen3-235B model card says that processing its full 1-million-token context requires approximately 1,000 GB of GPU memory, against 142 GB for the 4-bit weights, and advises reducing the context to 32,768 tokens on out-of-memory errors.
- Ollama's documentation states that parallel requests multiply the context: "a 2K context with 4 parallel requests will result in an 8K context and additional memory allocation".
- Ollama's K/V cache quantization setting (
OLLAMA_KV_CACHE_TYPE) reduces that memory:q8_0uses approximately half the memory of the defaultf16, andq4_0approximately a quarter.
The practical rule: size the card for the model plus the context you will really use, multiplied by the number of requests you serve in parallel. A coding agent with a 64,000-token context and four parallel sessions on a 20 GB model can outgrow a 24 GB card even though the model alone fits it. In vLLM the equivalent controls are --max-model-len and --gpu-memory-utilization; the vLLM guide for a 96 GB GPU walks through them.
Why does Ollama choose a different context on different cards?
Because its default now depends on how much VRAM it finds. Ollama's context-length documentation gives three tiers:
| VRAM detected | Default context | Cards in this guide |
|---|---|---|
| Under 24 GiB | 4k tokens | 16 GB cards; possibly 24 GB cards, which sit on the boundary |
| 24 to 48 GiB | 32k tokens | 24 GB and 32 GB cards |
| 48 GiB or more | 256k tokens | 80 GB and 96 GB cards; 48 GB cards sit on the boundary |
The same model can therefore behave differently on two servers: on a small card it silently forgets everything beyond 4,000 tokens, on a large one it reserves memory for a quarter of a million. Ollama's FAQ page still describes a 4,096-token default, so do not assume. Check what is loaded and where with ollama ps (the PROCESSOR column should read "100% GPU"), and set the value explicitly:
OLLAMA_CONTEXT_LENGTH=64000 ollama serve
Ollama recommends at least 64,000 tokens for web search, agents and coding tools. The Ollama and Open WebUI setup guide shows where to put the setting on a server that runs Ollama as a service.
What determines tokens per second?
Once a model fits, generation speed depends on the card's memory bandwidth (each token reads the active weights), the quantization, the context length, and how many requests the serving engine batches together. Bandwidth on the cards in this guide: 432 GB/s on the RTX PRO 4000 Blackwell, 1,792 GB/s on the RTX PRO 6000 Blackwell, 3.35 TB/s on the H100 SXM and 4.8 TB/s on the H200.
We publish no benchmarks of our own. Published third-party results, each valid only within its own setup:
- StorageReview (updated 3 November 2026; vLLM, Ministral 3 14B in FP8, 32 concurrent chat streams): L40S 995 output tokens per second, RTX PRO 6000 Blackwell Workstation Edition 2,283, H100 NVL 3,096.
- CloudRift (9 October 2025; vLLM, 400 concurrent requests, Qwen3-Coder-30B-A3B at AWQ 4-bit): RTX 4090 2,259 tokens per second, RTX 5090 4,570, RTX PRO 6000 8,425. Llama 3.3 70B at AWQ INT4 on one RTX PRO 6000: 1,031.
- Single stream: Database Mart reports about 185 tokens per second for gpt-oss 20b and about 134 for gpt-oss 120b with Ollama on an RTX PRO 6000 (from a search snippet of its benchmark page).
The concurrent figures are totals across all requests, which is what matters for an API endpoint. A single chat user sees the single-stream figure. None of these tests used the 300 W Max-Q edition of the RTX PRO 6000 that our servers carry, so treat them as an upper bound for it. To get your own number, send your real prompts at your real concurrency and record tokens per second and time to first token. A model that spills out of GPU memory is far slower than any difference between cards, which is why memory comes first. For what happens with no GPU at all, see LLM inference on CPUs.
One large card or several smaller ones?
One card that holds the whole model is simpler and usually faster than two that share it. Both common serving stacks say so in their own words. Ollama: "If the model will entirely fit on any single GPU, Ollama will load the model on that GPU", and only otherwise spreads it across all available GPUs. vLLM's guidance is not to distribute a model that fits one GPU, and to use tensor parallelism (--tensor-parallel-size 4) when it fits one node but not one card.
Several cards are the right answer in two situations. The first is a model larger than one card: qwen3:235b at 142 GB or qwen3-coder:480b at 290 GB need the 384 GB of a four-card server. The second is throughput: several copies of a model that fits one card, one per GPU, behind a load balancer. CloudRift measured 3,140 tokens per second for GLM-4.5-Air on one RTX PRO 6000 and 15,713 in total for eight replicas on eight cards. The multi-GPU vLLM guide covers both patterns on a four-card server.
The reverse also exists: one 96 GB RTX PRO 6000 can be divided with MIG into four isolated 24 GB instances, which suits several small models for separate teams. The RTX PRO 4000 and the RTX 5090 do not support MIG.
How to choose in five questions
- Which model, at which quantization? Read its size from the table. That is your floor.
- What context and how many parallel requests? Add generous headroom for long contexts and concurrency. If the sum is over about 20 GB, skip the 24 GB class.
- Is the licence compatible with your use? Apache 2.0 and MIT models carry no usage restrictions of the kind the non-commercial and revenue-capped licences do.
- How many hours a month will it run? For a few hours, rent by the hour or use an API. For an endpoint that runs all month, a fixed-price server is easier to budget: $449.99 for 24 GB, $1,656.15 for 96 GB, $5,966.44 for 384 GB, each with a one-time setup fee, on single-tenant hardware in an ISO 27001 certified data center.
- Who operates it? A dedicated server arrives with a clean Linux install and root access. Drivers, the serving stack, monitoring, and backups of anything you cannot re-download are yours to run, and a single machine has no built-in redundancy.
If the answers point to your own hardware, the LLM hosting page maps these models to the four MassiveGRID servers, and the RTX PRO 6000 vs H100 vs RTX 5090 comparison goes deeper on the cards themselves.
Frequently Asked Questions
What GPU do I need to run an LLM?
One with enough memory for the model and its context. A 24 GB card runs models up to about 32B at 4-bit (qwen3:32b and gemma4:31b are 20 GB on Ollama). A 96 GB card runs llama3.3:70b (43 GB) and gpt-oss:120b (65 GB). Models of 142 to 290 GB, such as qwen3:235b and qwen3-coder:480b, need about 384 GB across four cards.
How much VRAM does Llama 3.3 70B need?
The default 4-bit download on Ollama is 43 GB, the 8-bit version 75 GB and full 16-bit precision 141 GB, plus memory for context. It does not fit a 24 GB or 32 GB card. A 96 GB card holds the 4-bit or the 8-bit version on one GPU.
How much VRAM does gpt-oss 120b need?
65 GB as downloaded from Ollama (MXFP4 quantization). OpenAI's model card says it fits a single 80 GB GPU. On a 96 GB RTX PRO 6000 that leaves about 31 GB for context and a second small model.
Is 24 GB of VRAM enough for LLM inference in 2026?
For many internal uses, yes. The current 27B to 32B generation (qwen3.8:27b at 18 GB, gemma4:31b at 20 GB) fits with a moderate context, as do gpt-oss:20b and the 24B Mistral models. It is not enough for 70B-class models, long agent contexts with parallel sessions, or unquantized weights. Our 24 GB server is $449.99 a month and the 96 GB server $1,656.15.
Can any single server run the largest open models?
Not the 2026 flagships. Mistral Large 4 is 1,060.6 GB, DeepSeek V4-Pro 864.7 GB and Qwen3.8-2.4T 4,892 GB of weights on Hugging Face, all beyond a 384 GB four-card server. They need multi-node H200 or B200 deployments or the vendor's API.
LLM hosting on 24, 96 or 384 GB of GPU memory
MassiveGRID GPU dedicated servers cover the three memory classes in this guide: the RTX PRO 4000 with 24 GB for $449.99 per month, the RTX PRO 6000 with 96 GB from $1,656.15, and four RTX PRO 6000 with 384 GB for $5,966.44. Each is single-tenant bare metal with root access, a 1 Gbps port with unlimited traffic and a clean Linux install, ready for Ollama or vLLM.
See LLM HostingFurther Reading
- Ollama model library: download sizes, quantization and licences per tag
- Ollama documentation: context length defaults by VRAM
- Ollama FAQ: multi-GPU loading, parallel requests and K/V cache quantization
- Qwen3-235B-A22B model card: context length and GPU memory for 1M tokens
- OpenAI gpt-oss-120b model card: single 80 GB GPU
- vLLM documentation: when to use tensor parallelism
- StorageReview: enterprise GPU inference results with vLLM (updated 3 November 2026)
- CloudRift: benchmarking RTX GPUs for LLM inference (9 October 2025)
- AWS EC2 on-demand pricing (checked on 16 November 2026)
- Azure Retail Prices API (East US, Linux pay-as-you-go)
- Google Cloud accelerator-optimized machine pricing (G4)
- Oracle Cloud price list (GPU list prices per GPU-hour)
- RunPod pricing (Secure Cloud rates, checked on 16 November 2026)
- Lambda pricing (on-demand H100 instances)