LLM Hosting on Your Own GPU Server
LLM hosting means running an open-weight language model on hardware you control, behind your own API, instead of sending prompts to a model provider. MassiveGRID rents bare-metal GPU servers for it: 24 GB of vRAM for $449.99 a month, 96 GB from $1,656.15 and 384 GB for $5,966.44, each for one customer with root access. The four machines are listed on the GPU dedicated servers page; this page is about choosing the model, the server and the serving stack.
What is LLM hosting on a dedicated GPU server?
Self-hosted LLM hosting: you rent a server with a GPU, download the weights of an open-weight model, and run an inference server that speaks an OpenAI-compatible API. Your applications call that endpoint where they would have called a provider. Three things change compared with an API:
- Where the data goes. Prompts, documents and outputs are processed on a machine that only your organisation uses. No model provider receives them.
- How you pay. A flat monthly price for the hardware and nothing per token. Whether that costs less depends on volume, and the worked example below gives the break-even.
- Who operates it. You do. The server arrives with a clean operating system and root access; drivers, the serving stack, updates, monitoring and backups are yours to run.
The limit is memory. A model has to fit in the vRAM of the card, or of several cards, together with its context. That is why the choice of server starts from the model, not the other way round.
Which LLM fits which server?
Sizes are the downloads listed in the Ollama library on 16 November 2026, at the default 4-bit quantisation unless stated. Leave headroom above the weights: context and parallel requests use memory too.
| Model | Weights | Licence | Smallest server that fits | Monthly |
|---|---|---|---|---|
| gpt-oss 20B | 14 GB | Apache 2.0 | RTX PRO 4000, 24 GB | $449.99 |
| Qwen3.8 27B | 18 GB (8-bit 30 GB, 16-bit 56 GB) | Apache 2.0 | 24 GB at 4-bit; 96 GB for 8-bit or 16-bit | $449.99 |
| Gemma 4 31B | 20 GB (16-bit 63 GB) | Apache 2.0 | 24 GB, tight; 96 GB for long context | $449.99 |
| DeepSeek R1 distills 14B / 32B | 9.0 GB / 20 GB | MIT | 24 GB | $449.99 |
| Llama 3.3 70B | 43 GB (8-bit 75 GB) | Llama 3.3 Community License | RTX PRO 6000, 96 GB | $1,656.15 |
| DeepSeek R1 distill 70B | 43 GB | MIT | 96 GB | $1,656.15 |
| gpt-oss 120B | 65 GB | Apache 2.0 | 96 GB | $1,656.15 |
| Qwen3.5 122B | 81 GB | Apache 2.0 | 96 GB, about 15 GB left for context | $1,656.15 |
| Llama 3.3 70B at 16-bit | 141 GB | Llama 3.3 Community License | 4x RTX PRO 6000, 384 GB | $5,966.44 |
| Qwen3 235B | 142 GB (8-bit 250 GB) | Apache 2.0 | 384 GB | $5,966.44 |
| Qwen3-Coder 480B | 290 GB | Apache 2.0 | 384 GB | $5,966.44 |
Licences. Apache 2.0 and MIT models can be used commercially without a fee. The Llama licences are Meta's own terms, so read them before commercial use. Some well-known weights are not licensed for commercial use at all: Mistral Large 2411 is under the Mistral Research License and Command R+ under CC BY-NC 4.0. The DeepSeek R1 tags up to 70B are distills built on Qwen and Llama models; only the 671B tag is R1 itself.
What does not fit. The current open-weight flagships are larger than any of these servers: DeepSeek R1 and V3.1 at 671B are 404 GB even at 4-bit, Mistral Large 3 is 681.5 GB, DeepSeek V4-Pro 864.7 GB and Mistral Large 4 1,060.6 GB on Hugging Face. Mistral's model card for Large 3 names a node of eight H200s. Systems of that class are quoted by MassiveGRID through the H100 server page; otherwise these models are reached through an API. For the arithmetic behind the table see LLM vRAM requirements; for a single card in detail, the RTX PRO 6000 server and RTX PRO 4000 server pages.
Ollama or vLLM: which serving stack?
| Ollama | vLLM | |
|---|---|---|
| Best for | One team, a chat interface, trying models quickly | An API with many concurrent users |
| Install | One script, runs as a systemd service | Python package or the vllm/vllm-openai Docker image |
| Models | Pre-quantised tags from the Ollama library | Hugging Face repositories; AWQ, GPTQ and GGUF quantisation among others |
| Concurrency | One request at a time per model by default (OLLAMA_NUM_PARALLEL), queue of 512 | Continuous batching of incoming requests |
| API | Port 11434, bound to 127.0.0.1 by default | OpenAI-compatible on port 8000, API key with --api-key |
| Several GPUs | Spreads a model across GPUs when it does not fit on one | --tensor-parallel-size 4 for the four-card server |
Start with Ollama if the question is "which model is good enough for us". Move to vLLM when requests queue: Ollama's own FAQ says each parallel request multiplies the context memory, while vLLM is built to batch. The two stacks have their own pages: Ollama hosting and the AI inference server for vLLM, throughput and MIG.
Self-hosted LLM vs API: what does each cost?
A worked example with list prices read on 16 November 2026. Fifty people each make 100 requests a working day, 3,000 tokens in and 1,000 out. Over 22 working days that is 330 million input and 110 million output tokens a month. The break-even column is the monthly volume, at this 3:1 mix, above which the 96 GB server costs less.
| Option | Price per 1M tokens (in / out) | Monthly cost of the example | Break-even for the $1,656.15 server |
|---|---|---|---|
| OpenAI gpt-6-astra | $10.00 / $50.00 | $8,800.00 | About 83 million tokens a month |
| Claude Fable 5.2 | $8.00 / $40.00 | $7,040.00 | About 104 million |
| Claude Opus 5.5 | $4.00 / $20.00 | $3,520.00 | About 207 million |
| Gemini 3.1 Pro (preview, prompts up to 200,000 tokens) | $2.00 / $12.00 | $1,980.00 | About 368 million |
| OpenAI gpt-6.1-sol or Claude Sonnet 5.5 | $2.00 / $10.00 | $1,760.00 | About 414 million |
| Self-hosted on the RTX PRO 6000 server | none | $1,656.15 | - |
| Self-hosted on the RTX PRO 4000 server | none | $449.99 | - |
Read it carefully. The example is 440 million tokens a month, which is above every break-even in the table, so the flat price is lower than each of these list prices; against gpt-6-astra it is about a fifth of the bill. Below a model's break-even volume, paying per token costs less. The comparison is also not like for like: these are proprietary models and the server runs an open-weight one, so test whether a model that fits the card is good enough for your task, and whether one card carries your peak load, before you compare prices. Each server has a one-time setup fee ($1,656.15 and $346.48).
Per seat. Chat products are priced per user. A Standard seat of ChatGPT Business (as reported in search listings of OpenAI's pricing page) or Claude Team is $25 per user a month billed monthly, Microsoft 365 Copilot is $30 per user paid yearly on top of a Microsoft 365 licence, and GitHub Copilot is $19 (Business) or $39 (Enterprise) per seat. The 24 GB server costs the same as 18 seats at $25 and the 96 GB server about the same as 66; the private AI hosting page works that out for larger teams.
Where APIs win: low or unpredictable volume, the quality of frontier models that have no open weights, and zero operations. Where self-hosting wins: data that must not leave your control, steady high volume against premium-priced models, fine-tuned models of your own, and a bill that does not move with usage. Many teams run both: a private model for sensitive work and an API for the rest. The full method is in self-hosted LLM vs API cost: the break-even.
Four servers for self-hosted LLMs
Pick by the vRAM your model needs: 24 GB, 96 GB or 384 GB. Every server is bare metal for one customer, with root access, a 1 Gbps port with unlimited traffic and one dedicated IPv4, hosted in an ISO 27001 certified data center.
NVIDIA RTX PRO 4000
- 770 AI TOPS
- CUDA and FP4 support
- 1 dedicated IPv4 included
NVIDIA RTX PRO 6000
- 3,511 AI TOPS
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
NVIDIA RTX PRO 6000 Xeon
- 3,511 AI TOPS
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
4x NVIDIA RTX PRO 6000
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
Every RTX PRO server includes: a 1 Gbps port with unlimited traffic, one dedicated IPv4 address, hosting in an ISO 27001 certified data center, and your choice of Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS Stream, openSUSE or Arch Linux, or a rescue system to install your own OS. Billing runs monthly, quarterly, semi-annually, annually, biennially or triennially, and the setup fee is charged once with the first invoice.
Options on the RTX PRO 6000 Xeon: 512 GB DDR5 ECC with 2x 1.92 TB NVMe for $994.68/mo more, 768 GB DDR5 ECC with 4x 3.84 TB NVMe for $1,989.37/mo more, and a 10 Gbps uplink with 20 TB of traffic included for $71.29/mo more.
How to host your own LLM in five steps
- Choose the model, then the server. Take the size from the table above and add headroom for context. If you are unsure, start on 24 GB with an 18 GB model and move up when you hit the limit.
- Order and log in. Ubuntu 24.04 LTS is the default image. The server arrives clean, with root access and nothing pre-installed.
- Install the NVIDIA driver from NVIDIA's apt repository and reboot, then check
nvidia-smi:sudo apt install linux-headers-$(uname -r) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt update sudo apt install nvidia-open sudo reboot - Start a serving stack. Ollama for a quick start:
or vLLM (0.33.0 at the time of writing, Python 3.11 or newer) for an OpenAI-compatible API on port 8000:curl -fsSL https://ollama.com/install.sh | sh ollama run gpt-oss:20b
Run theuv venv --python 3.12 --seed --managed-python source .venv/bin/activate uv pip install vllm --torch-backend=auto vllm serve Qwen/Qwen3-32B --api-key "$VLLM_API_KEY" curl http://localhost:8000/v1/models -H "Authorization: Bearer $VLLM_API_KEY"curlcheck from a second terminal once the model has loaded. - Secure and measure. Keep the model port on localhost or behind a firewall, put a TLS reverse proxy with authentication in front, and measure tokens per second with your own prompts before you promise a response time to users.
A dedicated server is a single machine, so keep weights, adapters and configuration backed up off the server. Walkthroughs: drivers, CUDA and Docker in the first hour and a self-hosted LLM API with vLLM.
LLM hosting by task
Each page goes deeper on one part of the stack.
Ollama Hosting
Ollama and Open WebUI on a GPU server, with the library tags that fit.
vLLMAI Inference Server
Throughput, batching, MIG and monitoring for production APIs.
96 GBRTX PRO 6000 Server
The card, the three configurations and published benchmarks.
24 GBRTX PRO 4000 Server
The entry server for models up to about 20 GB.
384 GBMulti-GPU Server
Four cards for models that do not fit in 96 GB.
PrivacyPrivate AI Hosting
Confidential documents, regulated data and a model nobody else uses.
Guides for self-hosted LLMs
GPU Server Requirements for Local LLMs: VRAM by Model Size
How much VRAM a local LLM really needs: weight sizes by quantization, the KV cache formula, concurrency maths and which GPU fits which model.
GuideSelf-Hosted LLM vs API Cost: Where the Break-Even Really Is
Self-hosted LLM vs OpenAI API cost on 16 November 2026: a $1,656.15 GPU server breaks even at 414 million tokens a month against a $2/$10 model. Formula inside.
GuideBest GPU for LLM Inference in 2026: Pick by Model Size, Not by Name
Best GPU for LLM inference in 2026, ranked by use case: 24 GB runs models to 32B, 96 GB runs 70B and 120B, with a VRAM table for 30 current open models.
GuidevLLM on One 96 GB GPU: Serving Llama 3.3 70B, Qwen, gpt-oss-120b and DeepSeek Distills
vLLM serve Llama 3.3 70B on one 96 GB GPU: the 141.1 GB original does not fit, a 39.8 GB AWQ build does. Flags, API key, curl test, systemd unit.
GuideBuild a RAG Stack on Your Own Server: Postgres, pgvector and an LLM
Build a self-hosted RAG stack: pgvector setup, HNSW vs IVFFlat, chunking that decides quality, hybrid search, reranking, sizing and the real failure modes.
GuideFine-Tuning an Open Model on a Single GPU Server
Fine-tune an LLM on one GPU: why LoRA and QLoRA fit, sequence length as the real limit, a working PEFT config, data quality, when to stop, and merge vs adapter.
Frequently Asked Questions
Host your own LLM this week
Pick 24, 96 or 384 GB of vRAM, order online and run an open-weight model behind your own API. Flat monthly price, no per-token fee.