The best GPU for LLM inference is the smallest card that holds your model and its context entirely in GPU memory. In November 2026 that means 24 GB for open models up to about 32 billion parameters at 4-bit quantization (18 to 20 GB), 96 GB for 70B-class models (43 GB) and 120B mixture-of-experts models (65 to 87 GB), and 384 GB across four cards for 235B to 480B models (142 to 290 GB). Speed comes second and is mostly a matter of memory bandwidth.

This guide ranks GPUs by use case rather than naming one winner, then gives the memory each current open model needs, with sizes read from the Ollama library and Hugging Face on 16 November 2026. For the underlying arithmetic of parameters, bits and context, see our LLM VRAM requirements guide. MassiveGRID rents three of the configurations discussed as GPU dedicated servers; the others are included because they are sometimes the better answer.

What is the best GPU for LLM inference in 2026?

It depends on what you run and how often. Six cases cover most buyers.

Use caseBest fitGPU memoryExample on-demand price, checked on 16 November 2026
Private assistant for a small team, models up to 32BRTX PRO 4000 Blackwell, or a cloud L4 or A10G24 GB$449.99 a month as a MassiveGRID server; AWS g6.xlarge (L4) $0.8048 an hour
70B-class and 120B models, long context, several models loadedRTX PRO 6000 Blackwell96 GB$1,656.15 a month as a MassiveGRID server; RunPod Secure Cloud $2.69, AWS g7e.2xlarge $3.36312, Google Cloud g4-standard-48 $4.49993 and Azure NC144lds $5.50 an hour
Lowest latency per request, largest scaleH100 or H20080 GB or 141 GBH100: Lambda $3.79 (PCIe), AWS p5.4xlarge $6.88, Azure NC40ads_H100_v5 $6.98 and Oracle $10.00 an hour. H200: Oracle $10.00 an hour
Models of 100 to 300 GB4x RTX PRO 6000, or H200 and B200 nodes384 GB$5,966.44 a month as a MassiveGRID server; AWS g7e.24xlarge $16.57216 an hour
A developer's own deskGeForce RTX 509032 GBA workstation purchase; not on the large-cloud price lists we read
Occasional use, or under about 100 million tokens a monthNo GPU: an API or an hourly instance-Premium API list prices from $2.00 per million input tokens (gpt-6.1-sol, Claude Sonnet 5.5)

Three notes on the ranking. The 48 GB class (L40S, RTX 6000 Ada) is a cloud middle ground, but it holds a 43 GB model with almost no room for context, and the L40S has no MIG. The RTX 5090 is fast, yet NVIDIA's GeForce driver licence states "No Datacenter Deployment", so it is a workstation card. And the last row is serious: if your volume is below the break-even against API list prices, paying per token costs less.

The H100 class is the better choice for large-scale training and for very large models spread over many cards. H100, H200 and B200 systems are quoted by MassiveGRID sales rather than sold in the store.

How much VRAM do current open models need?

The table lists the default download size of each model on Ollama, which is 4-bit quantization (Q4_K_M) unless noted. The model needs that much GPU memory plus working memory for the context. "Yes" means the download leaves at least 10% of the memory free, "tight" means it loads with less than that, so only with a short context. The 384 GB column assumes the model is split across four 96 GB cards.

Model (Ollama tag)Default size24 GB96 GB384 GBLicence
llama3.1:8b4.9 GByesyesyesLlama 3.1 Community License
gemma4:12b8.0 GByesyesyesApache 2.0
phi4 (14B)9.1 GByesyesyesMIT
qwen3:14b9.3 GByesyesyesApache 2.0
gpt-oss:20b14 GB (MXFP4)yesyesyesApache 2.0
mistral-small3.2 (24B)15 GByesyesyesApache 2.0
devstral-small-2 (24B)15 GByesyesyesApache 2.0
gemma3:27b17 GByesyesyesGemma Terms of Use
qwen3.8:27b18 GByesyesyesApache 2.0
qwen3-coder:30b19 GByesyesyesApache 2.0
gemma4:31b20 GByesyesyesApache 2.0
qwen3:32b20 GByesyesyesApache 2.0
deepseek-r1:32b (a Qwen distill)20 GByesyesyesMIT
qwen3.6:35b23 GBtightyesyesApache 2.0
llama3.3:70b43 GBnoyesyesLlama 3.3 Community License
deepseek-r1:70b (a Llama distill)43 GBnoyesyesMIT
command-r-plus (104B)59 GB (Q4_0)noyesyesCC BY-NC 4.0, non-commercial
gpt-oss:120b65 GB (MXFP4)noyesyesApache 2.0
llama4:scout67 GBnoyesyesLlama 4 Community License
mistral-large (123B, 2411)73 GBnoyesyesMistral Research License, non-commercial
devstral-2 (123B)75 GBnoyesyesModified MIT, restricted above $20M monthly revenue
mistral-medium-3.5 (128B)80 GBnoyesyesModified MIT, exceptions for large-revenue companies
qwen3.5:122b81 GBnoyesyesApache 2.0
nemotron-3-super (120B)87 GBnotightyesNVIDIA model licence
qwen3:235b142 GBnonoyesApache 2.0
llama3.1:405b243 GBnonoyesLlama 3.1 Community License
llama4:maverick245 GBnonoyesLlama 4 Community License
qwen3-coder:480b290 GBnonoyesApache 2.0
deepseek-r1:671b, deepseek-v3.1:671b404 GBnononoMIT

Higher precision costs memory in proportion. qwen3.8:27b is 18 GB at 4-bit, 30 GB at 8-bit and 56 GB at full 16-bit precision; llama3.3:70b is 43, 75 and 141 GB. On a 96 GB card you can therefore run a 27B to 32B model unquantized or a 70B model at 8-bit, which a 24 GB card cannot do. Check the licence column before building a product: four of the larger models above are non-commercial or revenue-restricted.

Embedding models are small (nomic-embed-text 274 MB, bge-m3 1.2 GB) and sit beside a chat model on any of these cards.

Which flagship models do not fit even 384 GB?

The newest open-weight flagships of 2026 are far larger than any single server discussed here. Their weights on Hugging Face:

On Ollama these appear only as cloud tags, with no local download. Running them yourself means a multi-node H200 or B200 deployment, which is a quote-level project, or using the vendor's API. For most self-hosting the practical 2026 choices are the 27B to 35B generation (Qwen3.8, Gemma 4), the 70B class and the 120B mixture-of-experts models, all of which fit one 96 GB card.

How does context length change the memory you need?

The download size is the floor. Every token of context adds to a key-value cache that also lives in GPU memory, so a model that loads comfortably with a 4,000-token context can run out of memory at 128,000. The vendors' own statements show the scale:

The practical rule: size the card for the model plus the context you will really use, multiplied by the number of requests you serve in parallel. A coding agent with a 64,000-token context and four parallel sessions on a 20 GB model can outgrow a 24 GB card even though the model alone fits it. In vLLM the equivalent controls are --max-model-len and --gpu-memory-utilization; the vLLM guide for a 96 GB GPU walks through them.

Why does Ollama choose a different context on different cards?

Because its default now depends on how much VRAM it finds. Ollama's context-length documentation gives three tiers:

VRAM detectedDefault contextCards in this guide
Under 24 GiB4k tokens16 GB cards; possibly 24 GB cards, which sit on the boundary
24 to 48 GiB32k tokens24 GB and 32 GB cards
48 GiB or more256k tokens80 GB and 96 GB cards; 48 GB cards sit on the boundary

The same model can therefore behave differently on two servers: on a small card it silently forgets everything beyond 4,000 tokens, on a large one it reserves memory for a quarter of a million. Ollama's FAQ page still describes a 4,096-token default, so do not assume. Check what is loaded and where with ollama ps (the PROCESSOR column should read "100% GPU"), and set the value explicitly:

OLLAMA_CONTEXT_LENGTH=64000 ollama serve

Ollama recommends at least 64,000 tokens for web search, agents and coding tools. The Ollama and Open WebUI setup guide shows where to put the setting on a server that runs Ollama as a service.

What determines tokens per second?

Once a model fits, generation speed depends on the card's memory bandwidth (each token reads the active weights), the quantization, the context length, and how many requests the serving engine batches together. Bandwidth on the cards in this guide: 432 GB/s on the RTX PRO 4000 Blackwell, 1,792 GB/s on the RTX PRO 6000 Blackwell, 3.35 TB/s on the H100 SXM and 4.8 TB/s on the H200.

We publish no benchmarks of our own. Published third-party results, each valid only within its own setup:

The concurrent figures are totals across all requests, which is what matters for an API endpoint. A single chat user sees the single-stream figure. None of these tests used the 300 W Max-Q edition of the RTX PRO 6000 that our servers carry, so treat them as an upper bound for it. To get your own number, send your real prompts at your real concurrency and record tokens per second and time to first token. A model that spills out of GPU memory is far slower than any difference between cards, which is why memory comes first. For what happens with no GPU at all, see LLM inference on CPUs.

One large card or several smaller ones?

One card that holds the whole model is simpler and usually faster than two that share it. Both common serving stacks say so in their own words. Ollama: "If the model will entirely fit on any single GPU, Ollama will load the model on that GPU", and only otherwise spreads it across all available GPUs. vLLM's guidance is not to distribute a model that fits one GPU, and to use tensor parallelism (--tensor-parallel-size 4) when it fits one node but not one card.

Several cards are the right answer in two situations. The first is a model larger than one card: qwen3:235b at 142 GB or qwen3-coder:480b at 290 GB need the 384 GB of a four-card server. The second is throughput: several copies of a model that fits one card, one per GPU, behind a load balancer. CloudRift measured 3,140 tokens per second for GLM-4.5-Air on one RTX PRO 6000 and 15,713 in total for eight replicas on eight cards. The multi-GPU vLLM guide covers both patterns on a four-card server.

The reverse also exists: one 96 GB RTX PRO 6000 can be divided with MIG into four isolated 24 GB instances, which suits several small models for separate teams. The RTX PRO 4000 and the RTX 5090 do not support MIG.

How to choose in five questions

  1. Which model, at which quantization? Read its size from the table. That is your floor.
  2. What context and how many parallel requests? Add generous headroom for long contexts and concurrency. If the sum is over about 20 GB, skip the 24 GB class.
  3. Is the licence compatible with your use? Apache 2.0 and MIT models carry no usage restrictions of the kind the non-commercial and revenue-capped licences do.
  4. How many hours a month will it run? For a few hours, rent by the hour or use an API. For an endpoint that runs all month, a fixed-price server is easier to budget: $449.99 for 24 GB, $1,656.15 for 96 GB, $5,966.44 for 384 GB, each with a one-time setup fee, on single-tenant hardware in an ISO 27001 certified data center.
  5. Who operates it? A dedicated server arrives with a clean Linux install and root access. Drivers, the serving stack, monitoring, and backups of anything you cannot re-download are yours to run, and a single machine has no built-in redundancy.

If the answers point to your own hardware, the LLM hosting page maps these models to the four MassiveGRID servers, and the RTX PRO 6000 vs H100 vs RTX 5090 comparison goes deeper on the cards themselves.

Frequently Asked Questions

What GPU do I need to run an LLM?

One with enough memory for the model and its context. A 24 GB card runs models up to about 32B at 4-bit (qwen3:32b and gemma4:31b are 20 GB on Ollama). A 96 GB card runs llama3.3:70b (43 GB) and gpt-oss:120b (65 GB). Models of 142 to 290 GB, such as qwen3:235b and qwen3-coder:480b, need about 384 GB across four cards.

How much VRAM does Llama 3.3 70B need?

The default 4-bit download on Ollama is 43 GB, the 8-bit version 75 GB and full 16-bit precision 141 GB, plus memory for context. It does not fit a 24 GB or 32 GB card. A 96 GB card holds the 4-bit or the 8-bit version on one GPU.

How much VRAM does gpt-oss 120b need?

65 GB as downloaded from Ollama (MXFP4 quantization). OpenAI's model card says it fits a single 80 GB GPU. On a 96 GB RTX PRO 6000 that leaves about 31 GB for context and a second small model.

Is 24 GB of VRAM enough for LLM inference in 2026?

For many internal uses, yes. The current 27B to 32B generation (qwen3.8:27b at 18 GB, gemma4:31b at 20 GB) fits with a moderate context, as do gpt-oss:20b and the 24B Mistral models. It is not enough for 70B-class models, long agent contexts with parallel sessions, or unquantized weights. Our 24 GB server is $449.99 a month and the 96 GB server $1,656.15.

Can any single server run the largest open models?

Not the 2026 flagships. Mistral Large 4 is 1,060.6 GB, DeepSeek V4-Pro 864.7 GB and Qwen3.8-2.4T 4,892 GB of weights on Hugging Face, all beyond a 384 GB four-card server. They need multi-node H200 or B200 deployments or the vendor's API.

LLM hosting on 24, 96 or 384 GB of GPU memory

MassiveGRID GPU dedicated servers cover the three memory classes in this guide: the RTX PRO 4000 with 24 GB for $449.99 per month, the RTX PRO 6000 with 96 GB from $1,656.15, and four RTX PRO 6000 with 384 GB for $5,966.44. Each is single-tenant bare metal with root access, a 1 Gbps port with unlimited traffic and a clean Linux install, ready for Ollama or vLLM.

See LLM Hosting

Further Reading