Ollama is the fastest way to get a language model answering HTTP requests on your own server, and its defaults are built for a laptop rather than a machine with a public IP address. This guide covers the server-side decisions: whether you need a GPU at all, how to tune the service, how to expose the API without handing out free inference, and the point at which Ollama stops being the right tool.
Ollama is a wrapper around llama.cpp that adds three things: a model registry with one-line pulls, a REST API, and a systemd service. That packaging is why it spread so fast, and also why it is easy to deploy badly. Out of the box it listens only on localhost, holds a single model in memory for five minutes at a time, and ships no authentication whatsoever. Every one of those defaults is correct on a laptop and wrong on a server.
MassiveGRID infrastructure for local inference: Proxmox HA cluster with automatic failover · Ceph 3x replicated NVMe storage · independent CPU, RAM and storage scaling · 12 Tbps DDoS protection · deploy in any of 85+ metros across 30+ countries · 24/7 human support
Cloud VPS for CPU inference — from $1.99/mo
Dedicated VPS for consistent token throughput
GPU servers when CPU is no longer enough — A100 from $1,649/mo
CPU or GPU: Decide Before You Provision
Ollama runs on both, and the difference is not subtle. Text generation is bound by memory bandwidth, not raw compute, because every token requires reading the whole active model out of memory. A server DDR4 or DDR5 channel set delivers roughly 50 to 100 GB/s. An A100 delivers around 1.9 TB/s and an H100 around 3.4 TB/s.
That ratio shows up directly in generation speed. As a rough planning figure for an 8B model at 4-bit quantization:
| Target | Hardware | Generation speed | Honest use case |
|---|---|---|---|
| Experiments, batch jobs | 8 vCPU, 16 GB RAM | roughly 5–12 tokens/sec | Overnight summarisation, cron-driven classification |
| Internal tooling | 16 vCPU, 32 GB RAM | roughly 10–20 tokens/sec | A handful of colleagues, no concurrency |
| Interactive apps | GPU, 20–48 GB VRAM | 50–150+ tokens/sec | Chat UIs, agents, anything user-facing |
The practical rule: if a human waits for the output, use a GPU. If a queue waits for it, CPU is often fine and dramatically cheaper. Reading speed is around 5 to 8 tokens per second, so anything under that feels broken to a user.
Prerequisites
A clean Ubuntu 24.04 LTS server, root or sudo access, and a domain name pointed at the server if you intend to expose the API. For CPU inference, size RAM at roughly twice the model file size so the page cache can hold the weights. For GPU inference, install the NVIDIA driver first and confirm nvidia-smi reports the card before touching Ollama.
Storage matters more than people expect. Model weights are large and you will accumulate them: an 8B model at Q4 is around 4.7 GB, a 70B at Q4 is around 40 GB. Budget 100 GB of disk minimum if you plan to compare models.
Installation
The install script detects the platform, adds the systemd unit and creates a dedicated ollama user:
curl -fsSL https://ollama.com/install.sh | sh
systemctl status ollamaPiping a remote script into a shell deserves a moment of thought. If that is not acceptable in your environment, download it, read it, then run it:
curl -fsSL https://ollama.com/install.sh -o ollama-install.sh
less ollama-install.sh
sh ollama-install.shPull a model and confirm the service answers:
ollama pull llama3.1:8b
ollama list
curl http://127.0.0.1:11434/api/generate -d '{
"model": "llama3.1:8b",
"prompt": "Reply with the single word: ready",
"stream": false
}'Weights land in /usr/share/ollama/.ollama/models when Ollama runs as a service. That path is on the root filesystem by default, so if you attach a separate volume for models, move the directory and symlink it rather than fighting the service user's permissions later.
Server-Side Configuration
Ollama reads its configuration from environment variables, which on a systemd host belong in a drop-in file rather than the shipped unit:
systemctl edit ollama[Service]
Environment="OLLAMA_HOST=127.0.0.1:11434"
Environment="OLLAMA_KEEP_ALIVE=30m"
Environment="OLLAMA_MAX_LOADED_MODELS=2"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_FLASH_ATTENTION=1"Then reload and restart:
systemctl daemon-reload
systemctl restart ollamaWhat these actually do:
| Variable | Effect | Why you would change it |
|---|---|---|
OLLAMA_HOST | Bind address and port | Leave on localhost and terminate TLS in a reverse proxy. Setting 0.0.0.0 publishes an unauthenticated API |
OLLAMA_KEEP_ALIVE | How long an idle model stays resident | Default is 5 minutes. Longer avoids reload latency, at the cost of holding memory |
OLLAMA_MAX_LOADED_MODELS | Models resident at once | Raise only if total VRAM or RAM can hold them together |
OLLAMA_NUM_PARALLEL | Concurrent requests per model | Each slot reserves its own KV cache, so memory use scales with this number |
OLLAMA_FLASH_ATTENTION | Enables flash attention | Reduces KV cache memory and speeds up long contexts on supported GPUs |
The one that surprises people is OLLAMA_NUM_PARALLEL. Four parallel slots on a model with a 16k context does not cost one KV cache, it costs four. On a 20 GB card that can be the difference between running and an out-of-memory crash mid-request.
Exposing the API Safely
Ollama has no authentication, no rate limiting and no request logging worth the name. Anything reachable on the public internet will eventually be found and used to generate someone else's tokens on your hardware. Put nginx in front of it and require a key.
apt install nginx apache2-utils
htpasswd -c /etc/nginx/.ollama-users apiclientA minimal server block, with the long timeouts that generation requires:
server {
listen 443 ssl;
http2 on;
server_name llm.example.com;
ssl_certificate /etc/letsencrypt/live/llm.example.com/fullchain.pem;
ssl_certificate_key /etc/letsencrypt/live/llm.example.com/privkey.pem;
location / {
auth_basic "Inference API";
auth_basic_user_file /etc/nginx/.ollama-users;
proxy_pass http://127.0.0.1:11434;
proxy_set_header Host localhost:11434;
proxy_http_version 1.1;
proxy_buffering off;
proxy_read_timeout 600s;
proxy_send_timeout 600s;
}
}Two details are easy to miss. proxy_buffering off is what makes streamed responses arrive token by token instead of in one lump at the end. And overriding the Host header to localhost:11434 keeps Ollama's origin check happy, which otherwise rejects proxied requests.
Get the certificate with Let's Encrypt, and close the API port at the firewall so nothing bypasses the proxy:
certbot --nginx -d llm.example.com
ufw allow 443/tcp
ufw allow 22/tcp
ufw deny 11434/tcp
ufw enableThe OpenAI-Compatible Endpoint
Ollama exposes an OpenAI-shaped API at /v1, which means most existing SDKs and tools point at it with a base URL change and nothing else. This is the single most useful thing about running it:
curl https://llm.example.com/v1/chat/completions \
-u apiclient:yourpassword \
-H "Content-Type: application/json" \
-d '{
"model": "llama3.1:8b",
"messages": [{"role": "user", "content": "Summarise this log line: OOMKilled"}]
}'The compatibility is good but not total. Function calling, structured output modes and some sampling parameters behave differently or are ignored, so test the specific features your application depends on rather than assuming parity.
Choosing a Quantization
Ollama tags default to 4-bit quantization, usually Q4_K_M. That is a deliberate default and it is normally the right one, but it is worth knowing what you are trading:
| Quantization | Bytes per parameter | 8B model size | Quality impact |
|---|---|---|---|
| FP16 | 2.0 | ~16 GB | Reference quality |
| Q8_0 | ~1.0 | ~8.5 GB | Essentially indistinguishable |
| Q5_K_M | ~0.7 | ~5.7 GB | Very close to reference |
| Q4_K_M | ~0.6 | ~4.9 GB | Small measurable loss, rarely noticeable in practice |
| Q3_K_M | ~0.45 | ~4.0 GB | Visible degradation on reasoning tasks |
Below 4-bit, quality falls off faster than size does. If a model at Q3 fits and at Q4 does not, the better move is almost always a smaller model at Q4 rather than a bigger model at Q3.
Monitoring and Troubleshooting
Two commands answer most questions. ollama ps shows which models are resident and whether they are on GPU or CPU, which is how you catch a model that silently fell back to CPU because it did not fit in VRAM:
ollama ps
journalctl -u ollama -fOn a GPU host, watch utilisation and memory while a request runs:
nvidia-smi dmon -s umCommon failure modes and what they actually mean:
| Symptom | Usual cause |
|---|---|
| Sudden 10x slowdown | Model did not fit in VRAM and is running partly or wholly on CPU. Check ollama ps |
| First request slow, rest fast | Normal. Weights are being loaded. Raise OLLAMA_KEEP_ALIVE |
| Out of memory under load | OLLAMA_NUM_PARALLEL multiplied by context length exceeds available memory |
| Streaming arrives all at once | proxy_buffering is still on in nginx |
| 403 through the proxy | Origin check. Override the Host header as shown above |
When to Move Off Ollama
Ollama is excellent for a single user, a small team or a batch pipeline. It is not a serving stack. It does not do continuous batching, so concurrent requests queue rather than share a forward pass, and throughput per GPU is a fraction of what the hardware can do.
The signal to switch is concurrency. Once you have more than a few simultaneous users, or you are paying for a GPU and want the throughput you are paying for, move to vLLM, which batches requests continuously and manages KV cache in pages. Our guide to serving an LLM API with vLLM covers that migration.
Infrastructure That Does Not Drop Your Context
An inference endpoint tends to become load-bearing quickly. Something starts calling it on every page view, or a nightly job depends on it, and an hour of downtime turns into a queue of failed requests.
Every MassiveGRID server runs on a Proxmox high-availability cluster with automatic failover, so a hardware fault migrates the workload to another node instead of ending it. Storage is Ceph with three-way replication across independent NVMe drives, which matters when your model library and fine-tuned adapters live on that disk. Because CPU, RAM and storage scale independently, you can add the memory a larger model needs without paying for cores you will not use.
Start on a Cloud VPS to prove the workload out, move to a Dedicated VPS when consistent throughput matters, and step up to dedicated GPU infrastructure when a human is waiting on the output.