GPU Server Requirements for Local LLMs: VRAM by Model Size

Model weights are the predictable part of GPU sizing. The number that catches teams out is the KV cache, which grows with context length and again with every concurrent request, and which appears in almost none of the sizing tables published online. This guide gives you the arithmetic for all three consumers of VRAM, plus the mapping from model size to card.

Published sizing tables stop at weights because weights are static: a given model at a given quantization occupies a fixed number of bytes forever, and you can look it up once. The cache does not behave that way. It grows with every token of every open conversation, so the same deployment that runs comfortably in testing can exhaust the card the first time real traffic arrives.

MassiveGRID GPU configurations: A100 40 GB from $1,649/mo · A100 80 GB from $2,499/mo · H100 80 GB from $3,999/mo · RTX 6000 Ada 48 GB from $1,420.85/mo · multi-GPU with NVLink and InfiniBand on request

GPU cloud with a pre-configured AI stack
Dedicated GPU servers — RTX 4000 Ada, RTX 6000 Ada, H100 and above

The Three Things That Consume VRAM

Plan for all three or the arithmetic will not hold:

Weights. Parameter count multiplied by bytes per parameter. Fixed for a given model and quantization, and the easy part.

KV cache. Every token in every active request stores a key and value vector per layer. Grows linearly with context length and linearly with the number of concurrent requests.

Activations and runtime overhead. The CUDA context, the framework, temporary buffers during a forward pass. Budget one to two gigabytes plus roughly 10 percent of the weight size.

Weights by Model Size and Quantization

Bytes per parameter is what quantization changes. FP16 is two bytes, 8-bit is about one, and the common 4-bit formats land near 0.55 to 0.6 once you account for the higher-precision scales and outlier weights they keep:

Read that table with one number in mind: 80 GB, the capacity of an A100 80 GB or H100 80 GB. A 70B model does not fit in FP16 on a single card. It fits comfortably at 4-bit with room for a real KV cache. That single fact drives most deployment decisions.

Calculating the KV Cache

The formula is mechanical:

bytes per token = 2 x layers x kv_heads x head_dim x bytes_per_element

The leading 2 is for the key and the value. Note kv_heads, not attention heads. Modern models use grouped-query attention, where many query heads share one key-value head, and that distinction changes the answer by a factor of four or more.

Worked example, an 8B model with 32 layers, 8 key-value heads, head dimension 128, cache in FP16:

2 x 32 x 8 x 128 x 2 = 131,072 bytes per token = 128 KiB

So 8,000 tokens of context costs about 1 GB, and 32,000 tokens costs about 4 GB. Now multiply by concurrency, because each in-flight request holds its own cache:

This is the table that explains production incidents. An 8B model at 4-bit is under 5 GB of weights and looks trivial on a 48 GB card. Serve it at 32k context to 8 users and the cache alone wants 32 GB. The model was never the problem.

A 70B model has 80 layers, which pushes its cache to roughly 320 KiB per token, or 2.5 GB for 8k of context. Long context on large models is expensive in a way the weight table hides completely.

A Planning Formula That Works

Put the pieces together and add headroom:

VRAM = weights
     + (bytes_per_token x max_context x max_concurrent_requests)
     + 1.5 GB runtime
     + 15% headroom

The headroom is not padding for its own sake. Memory fragmentation, a burst of longer prompts than you planned for, and the allocator's own behaviour all consume real space. Provisioning to 100 percent of calculated need produces a service that works in testing and fails under traffic.

Serving frameworks give you a lever here. vLLM's --gpu-memory-utilization defaults to 0.9 and pre-allocates the rest of the card as a paged cache pool, which is why it reports high memory use immediately and why it handles bursts more gracefully than a naive allocator.

Mapping Models to GPUs

Assuming 4-bit weights, a 16k context and modest concurrency:

Two cards with the same VRAM are not interchangeable. The RTX 6000 Ada and the A100 80 GB both hold a 4-bit 70B, but the A100's HBM2e moves roughly twice the bytes per second of the RTX's GDDR6, and generation speed tracks bandwidth. For batch work where latency is irrelevant, the cheaper card is the better buy. For interactive serving, it is not.

When to Split Across GPUs

Tensor parallelism shards each layer across cards, so two 80 GB GPUs give you a usable 160 GB. It is not free: every forward pass requires collective communication between the cards, and on a PCIe-only host that interconnect becomes the bottleneck. NVLink or InfiniBand is what makes multi-GPU scale close to linearly, which is why it is worth specifying rather than assuming.

Prefer one card when the model fits. Split when it genuinely does not, or when you need aggregate throughput beyond what a single card delivers.

A Worked Sizing Example

Requirement: a 70B assistant, 16k context, up to 8 concurrent users.

weights (4-bit)        41 GB
KV cache               320 KiB x 16,384 x 8  =  40 GB
runtime                 1.5 GB
subtotal               82.5 GB
plus 15% headroom      95 GB

That does not fit on one 80 GB card, which is the useful result. Three options, in order of how often they are the right answer: cut the context to 8k and it fits with room to spare; cap concurrency at 4 and it fits; or move to two cards and stop worrying about either limit. Discovering this on a spreadsheet costs nothing. Discovering it in production costs an incident.

Dedicated GPUs, Not Time Slices

VRAM planning only holds if the card is actually yours. On shared or oversubscribed GPU platforms, another tenant's allocation changes what you can fit, and the failure looks like a random out-of-memory error you cannot reproduce.

MassiveGRID GPU instances are dedicated. The A100 40 GB tier ships with 16 vCPU, 120 GB of system RAM and a 512 GB NVMe volume; the A100 80 GB tier with 24 vCPU and 240 GB RAM; the H100 80 GB tier with 32 vCPU, 480 GB of DDR5 and 100 Gbps networking. System RAM matters more than people expect, because model loading, tokenisation and any retrieval layer all run on the CPU side.

Every instance sits on the same Proxmox HA cluster and Ceph triple-replicated NVMe storage as the rest of the platform, in any of the 85+ metros we deploy into. PyTorch, CUDA, cuDNN, Docker with the NVIDIA container runtime, Jupyter, Hugging Face libraries and vLLM come pre-installed, so sizing is the hard part and setup is not.

Compare GPU configurations, or read our GPU selection guide for the compute side of the same decision.