Ollama Hosting on a Dedicated GPU Server
Ollama hosting is a server that runs Ollama for you alone, so a team can chat with open-weight models without sending prompts to a provider. On MassiveGRID that is a bare-metal GPU server: 24 GB of vRAM for $449.99 a month, enough for tags up to about 20 GB such as gpt-oss:20b and qwen3.8:27b, or 96 GB for $1,656.15 for llama3.3:70b and gpt-oss:120b. Both are on the GPU dedicated servers page, and small models do not need a GPU at all, as explained below.
What is Ollama hosting?
Ollama is a model runner: one command pulls a quantised model and serves it over a local HTTP API. Ollama hosting is running it on a server in a data center, reachable by your team around the clock, where a laptop would sleep and a workstation would sit under a desk. On a dedicated GPU server the whole card is yours, so a model stays loaded and answers at GPU speed.
The defaults below are from the Ollama documentation, read on 16 November 2026 (current release v0.40.16). They explain most of the surprises people meet on day one.
| Setting | Default | What it means on a server |
|---|---|---|
| Bind address | 127.0.0.1, port 11434 | Only the server itself can reach the API. Change it with OLLAMA_HOST |
| Parallel requests per model | 1 (OLLAMA_NUM_PARALLEL) | A second user waits in the queue (512 requests by default) until the first answer finishes |
| Keep-alive | 5 minutes (OLLAMA_KEEP_ALIVE) | An idle model is unloaded; the next request pays the load time again |
| Loaded models | 3 per GPU (OLLAMA_MAX_LOADED_MODELS) | Several models can stay in vRAM if each fits completely |
| Context length | By vRAM: under 24 GiB 4k, 24 to 48 GiB 32k, 48 GiB and above 256k | A 24 GB card sits on a boundary: check with ollama ps and set OLLAMA_CONTEXT_LENGTH yourself |
Ollama's documentation adds that coding tools, agents and web search "should be set to at least 64000 tokens", and that a larger context raises the memory a model needs. Parallelism does the same: required memory scales with OLLAMA_NUM_PARALLEL multiplied by the context length.
Which Ollama models fit each GPU server?
Sizes are the downloads printed on the Ollama library tag pages on 16 November 2026, at the default quantisation. The download is the floor: add memory for context before deciding that a tag fits.
| Ollama tag | Size | Licence | RTX PRO 4000, 24 GB | RTX PRO 6000, 96 GB |
|---|---|---|---|---|
| llama3.1:8b | 4.9 GB | Llama 3.1 Community License | Fits | Fits |
| gemma4:12b | 8.0 GB | Apache 2.0 | Fits | Fits |
| deepseek-r1:14b | 9.0 GB | MIT | Fits | Fits |
| gpt-oss:20b | 14 GB | Apache 2.0 | Fits | Fits |
| mistral-small3.2:24b | 15 GB | Apache 2.0 | Fits | Fits |
| qwen3.8:27b | 18 GB | Apache 2.0 | Fits, about 6 GB spare | Fits, 8-bit (30 GB) too |
| qwen3-coder:30b | 19 GB | Apache 2.0 | Fits, about 5 GB spare | Fits |
| gemma4:31b, deepseek-r1:32b | 20 GB | Apache 2.0, MIT | Tight | Fits |
| llama3.3:70b, deepseek-r1:70b | 43 GB | Llama 3.3 Community License, MIT | No | Fits, 8-bit (75 GB) too |
| gpt-oss:120b | 65 GB | Apache 2.0 | No | Fits |
| qwen3.5:122b | 81 GB | Apache 2.0 | No | Fits with short context |
| nomic-embed-text, bge-m3 | 274 MB, 1.2 GB | Apache 2.0, MIT | Fits beside a chat model | Fits |
Monthly prices: $449.99 for the 24 GB server and $1,656.15 for the 96 GB server. Tags above 96 GB, such as qwen3:235b at 142 GB, need the four-card server at $5,966.44: Ollama's FAQ says a model that does not fit on one GPU "will be spread across all the available GPUs". The LLM hosting page compares the models themselves; the RTX PRO 4000 server page covers the 24 GB card.
Open WebUI hosting: a chat interface for your team
Ollama is an API. Most teams add Open WebUI (v0.13.3 at the time of writing) for a browser chat with accounts and history. With Docker installed, the project's README gives this command, which serves the interface on port 3000 of the server:
docker run -d -p 3000:8080 --add-host=host.docker.internal:host-gateway -v open-webui:/app/backend/data --name open-webui --restart always ghcr.io/open-webui/open-webui:mainIf Ollama listens somewhere other than the default, add -e OLLAMA_BASE_URL=http://your-ollama-server:11434. Without Docker, pip install open-webui and open-webui serve run it on port 8080 with Python 3.12.
Licence note. Since v0.6.6 Open WebUI is published under its own Open WebUI License, not an OSI open-source licence: the branding must stay in place unless the deployment has 50 or fewer users in a 30-day period, or you hold written permission or an enterprise licence. Check it before rebranding the interface for a larger organisation.
Security. Leave Ollama on 127.0.0.1. If you set OLLAMA_HOST=0.0.0.0, anyone who can reach port 11434 can send requests to it, so never publish that port to the internet without authentication in front of it. Expose only the web interface, through a reverse proxy with TLS, restrict the rest with a firewall, and reach the raw API over SSH or a VPN. The full build is in Ollama and Open WebUI as a private ChatGPT.
Do you need a GPU server for Ollama?
Not always, and it would be dishonest to say otherwise. Ollama runs on CPU, and a small model on a CPU server costs a fraction of a GPU machine.
| Cloud VPS (CPU only) | GPU server with Ollama | GPU server with vLLM | |
|---|---|---|---|
| Fits | Small tags such as qwen3:4b (2.5 GB) or llama3.1:8b (4.9 GB) in system RAM | Tags up to about 20 GB on 24 GB, up to about 81 GB on 96 GB | The same cards, models from Hugging Face |
| Speed | Slow: fine for background jobs and one patient user | GPU speed, one request at a time per model by default | Batches many concurrent requests |
| Price | Cloud VPS from $3.99 a month, sized by the RAM the model needs | $449.99 or $1,656.15 a month | The same servers |
| Choose it when | Classification, summaries overnight, a personal assistant, testing prompts | A team that chats, a coding assistant, models from 14 GB upwards | An application or API with many users at once |
Where the VPS wins: price, whenever the model is small and nobody is waiting for the answer. Start there with running Ollama on a VPS and what to expect from CPU inference. Where the GPU wins: response time, and every model too large for a few gigabytes of RAM. When to move from Ollama to vLLM: when requests queue behind each other. Raising OLLAMA_NUM_PARALLEL multiplies the context memory, while vLLM is designed around continuous batching. The AI inference server page picks up from there.
Two GPU servers for Ollama
24 GB for tags up to about 20 GB, 96 GB for the 70B and 120B tags. Bare metal for one customer, root access, a 1 Gbps port with unlimited traffic and one dedicated IPv4, in an ISO 27001 certified data center.
NVIDIA RTX PRO 4000
- 770 AI TOPS
- CUDA and FP4 support
- 1 dedicated IPv4 included
NVIDIA RTX PRO 6000
- 3,511 AI TOPS
- CUDA, FP4 and Multi-Instance GPU (MIG) support
- 1 dedicated IPv4 included
Every RTX PRO server includes: a 1 Gbps port with unlimited traffic, one dedicated IPv4 address, hosting in an ISO 27001 certified data center, and your choice of Ubuntu, Debian, AlmaLinux, Rocky Linux, CentOS Stream, openSUSE or Arch Linux, or a rescue system to install your own OS. Billing runs monthly, quarterly, semi-annually, annually, biennially or triennially, and the setup fee is charged once with the first invoice.
Install Ollama on the GPU server in four steps
The server arrives with a clean operating system and root access; Ollama and the NVIDIA driver are not pre-installed. On Ubuntu 24.04 LTS:
- Install the NVIDIA driver (Ollama needs driver 550 or newer) and reboot:
Then runsudo apt install linux-headers-$(uname -r) wget https://developer.download.nvidia.com/compute/cuda/repos/ubuntu2404/x86_64/cuda-keyring_1.1-1_all.deb sudo dpkg -i cuda-keyring_1.1-1_all.deb sudo apt update sudo apt install nvidia-open sudo rebootnvidia-smi: it should list the card and its memory. - Install Ollama. The script sets up a systemd service called
ollama:curl -fsSL https://ollama.com/install.sh | sh - Pull and test a model, then confirm it runs on the GPU:
The PROCESSOR column should read "100% GPU". If it shows a CPU share, the model and its context do not fit in vRAM.ollama run gpt-oss:20b ollama ps - Set the server defaults. Run
sudo systemctl edit ollamaand add the variables you need under[Service], for example:
Then[Service] Environment="OLLAMA_CONTEXT_LENGTH=64000" Environment="OLLAMA_KEEP_ALIVE=24h"sudo systemctl daemon-reloadandsudo systemctl restart ollama. Logs are injournalctl -e -u ollama.
One server is one machine: keep your Modelfiles, Open WebUI data volume and any fine-tuned adapters backed up somewhere else. Models from the library can simply be pulled again.
Beyond Ollama
Where to go when the workload grows.
LLM Hosting
Model-to-server table, licences and API cost compared.
vLLMAI Inference Server
For concurrent users: vLLM, batching and MIG.
24 GBRTX PRO 4000 Server
The entry GPU server in detail.
96 GBRTX PRO 6000 Server
For llama3.3:70b, gpt-oss:120b and long context.
PrivacyPrivate AI Hosting
A private assistant for confidential work.
Ollama guides
Ollama and Open WebUI Setup: A Private ChatGPT Alternative on Your Own GPU Server
Ollama and Open WebUI setup in 5 steps: install Ollama, pick a model for 24 GB or 96 GB of VRAM, run Open WebUI in Docker and add TLS with Caddy.
GuideHow to Run Ollama on a VPS: Self-Hosted LLM Inference
Run Ollama on a VPS for self-hosted LLM inference: CPU vs GPU sizing, systemd tuning, nginx auth, the OpenAI-compatible API and when to move to vLLM.
GuideCPU-Only LLM Inference: What Actually Works Without a GPU
CPU LLM inference: why bandwidth not compute is the limit, which workloads run well, llama.cpp setup, quantisation, sizing and the honest GPU comparison.
GuideThe First Hour on a GPU Server: NVIDIA Drivers, CUDA and Docker on Ubuntu 24.04
Install NVIDIA drivers on Ubuntu 24.04 server in 7 steps: nvidia-open from NVIDIA's repo, Docker, NVIDIA Container Toolkit 1.20.1 and a test container.
Guiden8n and Flowise with a Local LLM: Ollama or vLLM on Your Own GPU Server Instead of an API
n8n local LLM setup: point the Ollama or OpenAI Chat Model node, or Flowise ChatOllama, at your own GPU server over TLS with an API key. Costs from $449.99.
GuideGPU Server Requirements for Local LLMs: VRAM by Model Size
How much VRAM a local LLM really needs: weight sizes by quantization, the KV cache formula, concurrency maths and which GPU fits which model.
Frequently Asked Questions
Run Ollama on a GPU of your own
A 24 GB server for $449.99 a month or 96 GB for $1,656.15. Install Ollama with one command and give your team a private chat.