An Ollama and Open WebUI setup on a GPU server takes five steps: install Ollama with its one-line script, pull a model that fits the card (14 GB gpt-oss:20b on 24 GB of VRAM, 65 GB gpt-oss:120b or 43 GB llama3.3:70b on 96 GB), start Open WebUI with its documented Docker command, put Caddy in front for HTTPS, and create the admin account. The result is a private, self-hosted ChatGPT alternative: your team signs in at your own domain and no prompt leaves the server.
Two defaults decide whether it feels good for more than one person: Ollama processes one request per model at a time (OLLAMA_NUM_PARALLEL defaults to 1) and unloads an idle model after 5 minutes. This guide sets both, along with context length, and explains the Open WebUI licence and where a hosted assistant is still the better choice.
What you are building
Three services on one server, each with one job, and only one of them reachable from the internet:
| Component | Job | Listens on | Reachable from the internet |
|---|---|---|---|
| Ollama (v0.40.16 on 16 November 2026) | Downloads models, loads them on the GPU, answers API calls | 127.0.0.1:11434 (its default) | No |
| Open WebUI (v0.13.3 on 16 November 2026) | The chat interface, user accounts, chat history, document upload | Port 8080 on the host, closed in the firewall | No |
| Caddy | TLS certificate and HTTPS, forwards to Open WebUI | Ports 80 and 443 | Yes |
Prompts, uploaded files and chat history stay on the server: nothing is sent to a model provider. You are also the operator: updates, backups of the chat database and access control are yours.
The guide assumes a server in the state the first-hour setup guide leaves it in: Ubuntu 24.04, nvidia-smi working, Docker installed, ufw allowing only SSH. You also need a DNS name (for example chat.example.com) with an A record pointing at the server's IPv4 address. It works on any of the GPU dedicated servers; the model you can run depends on the card, 24 GB on the RTX PRO 4000 or 96 GB on the RTX PRO 6000.
Step 1: install Ollama and confirm it sees the GPU
Ollama documents a single command for Linux. It installs the binary and a systemd service named ollama that runs as its own user:
curl -fsSL https://ollama.com/install.sh | shCheck:
ollama -v
sudo systemctl status ollama
nvidia-smiThe service should be active. Ollama "binds 127.0.0.1 port 11434 by default", so it is not exposed; leave it that way. The log is journalctl -e -u ollama.
Step 2: which model fits a 24 GB or a 96 GB card?
A model has to fit in GPU memory together with its context (the conversation it is holding). The download size on the Ollama library is the floor; the context is added on top and grows with its length and with the number of parallel requests. The sizes below are the default tags as the Ollama library listed them on 16 November 2026; defaults are 4-bit quantised builds (Q4_K_M) unless noted.
| Model tag | Download | Context window listed | Licence | 24 GB card | 96 GB card |
|---|---|---|---|---|---|
gemma4:12b | 8.0 GB | 256K | Apache 2.0 | Yes | Yes |
qwen3:14b | 9.3 GB | 40K | Apache 2.0 | Yes | Yes |
gpt-oss:20b (MXFP4) | 14 GB | 128K | Apache 2.0 | Yes | Yes |
mistral-small3.2 (24b) | 15 GB | 128K | Apache 2.0 | Yes, with a short context | Yes |
qwen3.8:27b | 18 GB | 256K | Apache 2.0 | Tight: 6 GB left for context | Yes |
gemma4:31b | 20 GB | 256K | Apache 2.0 | Tight: 4 GB left for context | Yes |
llama3.3:70b | 43 GB | 128K | Llama 3.3 Community License | No | Yes |
gpt-oss:120b (MXFP4) | 65 GB | 128K | Apache 2.0 | No | Yes |
Two practical choices: on the 24 GB card start with gpt-oss:20b, which OpenAI's model card says runs within 16 GB of memory; on the 96 GB card start with gpt-oss:120b (the card says it fits a single 80 GB GPU) or llama3.3:70b, which leaves 53 GB for context and a second small model. For how these numbers are derived, see how much VRAM an LLM needs.
ollama pull gpt-oss:20b
ollama run gpt-oss:20b "Summarise what a reverse proxy does in two sentences."
ollama psCheck: the model answers, and ollama ps shows it loaded with 100% GPU in the PROCESSOR column. If the column shows a CPU share, the model plus its context did not fit in VRAM: pick a smaller model or a shorter context.
Step 3: set context length, parallel requests and keep-alive for a team
Ollama's defaults are tuned for one person at a terminal. Four settings decide how it behaves with several users, and all four are environment variables on the service.
OLLAMA_CONTEXT_LENGTH. Ollama's context-length page says the default depends on VRAM: 4k below 24 GiB, 32k from 24 to 48 GiB, and 256k at 48 GiB or more (the FAQ still quotes 4096 tokens, so do not rely on the default; set it). The same page advises at least 64,000 tokens for web search, agents and coding tools, and warns that a larger context increases the memory a model needs.OLLAMA_NUM_PARALLEL. The maximum number of requests each model processes at the same time. The default is 1, so by default a second user waits in the queue until the first answer is finished. The documentation is explicit about the cost of raising it: required memory scales byOLLAMA_NUM_PARALLELtimesOLLAMA_CONTEXT_LENGTH. Four parallel requests at a 16,000-token context allocate a 64,000-token context.OLLAMA_KEEP_ALIVE. By default a model is unloaded after 5 minutes without requests, and the next user waits while it loads again. On a server dedicated to this job, keep it loaded:-1means indefinitely.OLLAMA_MAX_LOADED_MODELS. Default 3 per GPU. Each loaded model must fit completely in VRAM, so on a 24 GB card set it to 1 to stop users filling the card by switching models.
Set them with a systemd override, as Ollama's Linux page documents:
sudo systemctl edit ollamaIn the editor, add:
[Service]
Environment="OLLAMA_CONTEXT_LENGTH=16000"
Environment="OLLAMA_NUM_PARALLEL=4"
Environment="OLLAMA_KEEP_ALIVE=-1"
Environment="OLLAMA_MAX_LOADED_MODELS=1"sudo systemctl daemon-reload
sudo systemctl restart ollama
ollama run gpt-oss:20b "ready"
ollama psCheck: ollama ps still reports 100% GPU and nvidia-smi shows memory in use below the card's total. If not, lower the parallel count or the context until it does. The values above are a starting point for the 24 GB card; on 96 GB you have room for a longer context.
Step 4: run Open WebUI with its documented Docker command
Open WebUI's quick start publishes the container with -p 3000:8080 and reaches Ollama through host.docker.internal. On this server that has two problems: Ollama listens only on 127.0.0.1, which a bridged container cannot reach, and a published Docker port bypasses ufw. Open WebUI's connection troubleshooting page documents the variant that solves both, host networking with Ollama addressed on the loopback interface:
docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 --name open-webui --restart always ghcr.io/open-webui/open-webui:mainBefore you run it, add one option from the quick start. Open WebUI's documentation says that without a fixed WEBUI_SECRET_KEY every recreated container logs everyone out, and tells you to generate one with openssl rand -hex 32. The complete command:
openssl rand -hex 32
docker run -d --network=host -v open-webui:/app/backend/data -e OLLAMA_BASE_URL=http://127.0.0.1:11434 -e WEBUI_SECRET_KEY=your-secret-key --name open-webui --restart always ghcr.io/open-webui/open-webui:mainReplace your-secret-key with the output of the first command and store it somewhere safe. With host networking the interface is on port 8080 of the host, and because the process binds on the host rather than through a published port, ufw applies: port 8080 stays closed to the internet. For a production deployment, pin a release tag such as v0.13.3 in place of main, so an image pull never changes the version under you.
Check:
docker ps --filter name=open-webui
curl -sI http://127.0.0.1:8080 | head -n 1The first start takes about a minute. The second command should return an HTTP 200 line. Everything Open WebUI stores (accounts, chats, uploaded documents) is in the Docker volume open-webui, which is the thing to back up off the server.
Step 5: put TLS in front with Caddy and create the admin account
Passwords and prompts must not cross the internet in clear text, so nobody connects to port 8080 directly. Caddy obtains and renews a certificate automatically when it knows the hostname. Install it from Caddy's apt repository, as its install page documents for Debian and Ubuntu:
sudo apt install --yes debian-keyring debian-archive-keyring apt-transport-https curl
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/gpg.key' | sudo gpg --dearmor -o /usr/share/keyrings/caddy-stable-archive-keyring.gpg
curl -1sLf 'https://dl.cloudsmith.io/public/caddy/stable/debian.deb.txt' | sudo tee /etc/apt/sources.list.d/caddy-stable.list
sudo chmod o+r /usr/share/keyrings/caddy-stable-archive-keyring.gpg
sudo chmod o+r /etc/apt/sources.list.d/caddy-stable.list
sudo apt update
sudo apt install caddyThe package runs Caddy as a systemd service that reads /etc/caddy/Caddyfile. Replace the file's contents with your hostname and one directive:
chat.example.com
reverse_proxy :8080Caddy's documentation says that for a public domain name it will attempt to get a publicly trusted certificate, and that your DNS records must point to the machine and ports 80 and 443 must be open. Open them and reload:
sudo ufw allow 80
sudo ufw allow 443
sudo systemctl reload caddyCheck: curl -I https://chat.example.com returns a response over HTTPS with a valid certificate, and sudo ufw status lists only 22, 80 and 443. nginx works too: a certificate on 443, proxy_pass to 127.0.0.1:8080 and WebSocket upgrade headers.
Now open https://chat.example.com in a browser. Do this immediately: the first screen offers Create Admin Account, and Open WebUI's documentation states that this first account is the administrator. Whoever reaches the page first owns the instance. Sign-up switches itself off once the admin account exists. To let colleagues in, either create their accounts in the Admin Panel or turn on new sign-ups under Settings, Authentication; new accounts then wait as Pending until you approve them.
How many users can one card serve?
There is no single number: it depends on the model, the length of prompts and answers, how many people send a message in the same second (far fewer than the number of accounts) and OLLAMA_NUM_PARALLEL. What the documentation supports:
- With the default of 1 parallel request, users are served one after another. Ollama queues up to 512 requests (
OLLAMA_MAX_QUEUE) before rejecting more, so nothing fails, but answers wait. - Each extra parallel slot costs context memory. A 24 GB card with a 14 GB model has about 10 GB for context across all slots; a 96 GB card with a 65 GB model has about 31 GB, and with a 43 GB model about 53 GB.
- Published single-stream figures give an order of magnitude for one user at a time. Database Mart reported about 185 tokens per second for gpt-oss 20B and about 134 for gpt-oss 120B with Ollama on an RTX PRO 6000 (a third-party figure from a search-result summary of their benchmark page, checked on 16 November 2026, not a MassiveGRID measurement).
Measure your own case: run ollama run gpt-oss:20b --verbose with a prompt typical of your team and read the eval rate it prints in tokens per second. Then send prompts from four browser sessions at once and watch the waiting time. If people queue, raise the parallel count while ollama ps still shows 100% GPU.
When dozens of people or an application send requests at the same moment, a batching server uses the same card more efficiently; see serving Llama, Qwen and DeepSeek models with vLLM on a 96 GB GPU. Open WebUI can talk to any OpenAI-compatible endpoint, so you can change the engine later without changing what users see.
The Open WebUI licence, and where a hosted assistant is the better choice
Licence. Ollama is MIT-licensed software, and each model has its own licence. Open WebUI is different: since v0.6.6 (19 April 2025) it is released under the "Open WebUI License", which is not an OSI open-source licence. Code up to v0.6.5 remains BSD-3-Clause. The new licence keeps the software free to use, modify and redistribute, but you may not alter, remove or obscure the Open WebUI branding unless your deployment has 50 or fewer users in a 30-day period, you have written permission, or you hold an enterprise licence. If you planned to rebrand it for more than 50 users, read the licence page first.
Where a hosted assistant wins. A self-hosted chat on open-weight models is not a like-for-like replacement for ChatGPT, Claude or Gemini:
- Model quality. The frontier models from the API vendors are ahead of what fits on one card, most visibly on long reasoning chains and difficult code.
- Low volume. For a handful of people who send a few messages a day, paying per seat or per token costs less than a server at $449.99 or $1,656.15 per month. The break-even comparison between a self-hosted LLM and an API shows where the lines cross.
- Operations. A hosted product leaves you no drivers, backups or certificates to look after.
The self-hosted route wins when prompts and documents must not leave infrastructure you control, when usage is heavy enough that a fixed monthly bill is lower than per-token charges, or when you need a specific open model, a fine-tune or no rate limits. The Ollama hosting page maps team sizes and models to the four servers.
What goes wrong
| Symptom | Cause | Fix |
|---|---|---|
| Open WebUI loads but the model selector is empty | The container cannot reach Ollama. With the quick-start command (-p 3000:8080) Ollama on 127.0.0.1 is unreachable from the bridge network | Use the host-networking command from step 4 with OLLAMA_BASE_URL=http://127.0.0.1:11434. Test from the host with curl http://127.0.0.1:11434/api/tags. |
Answers are slow and ollama ps shows a CPU percentage | Model plus context does not fit in VRAM, so part of it runs on the CPU | Lower OLLAMA_CONTEXT_LENGTH or OLLAMA_NUM_PARALLEL, or pull a smaller model. |
| The first message after a pause takes much longer | The model was unloaded after the 5-minute default | Set OLLAMA_KEEP_ALIVE=-1 as in step 3. |
| The reply stays empty or hangs behind the proxy | WebSocket traffic is not passing through the reverse proxy | Caddy's reverse_proxy handles WebSockets without extra configuration; with nginx add the upgrade headers. Open WebUI's troubleshooting page covers the CORS origin setting. |
| Everyone is logged out after an update | The container was recreated without a fixed WEBUI_SECRET_KEY | Pass the same key on every docker run. |
| Caddy cannot obtain a certificate | DNS does not point at the server yet, or ports 80 and 443 are closed | Check the A record with dig +short chat.example.com and the firewall with sudo ufw status; read journalctl -u caddy. |
Security recap. Ollama on 127.0.0.1 only, because its API has no authentication of its own; Open WebUI on a port the firewall keeps closed; HTTPS as the only public entry; sign-ups off or approval required; the open-webui volume backed up off the server.
Frequently Asked Questions
Can I run Ollama and Open WebUI on the same GPU server?
Yes, and it is the usual layout. Ollama runs as a systemd service on 127.0.0.1:11434 and uses the GPU; Open WebUI runs in a Docker container and needs no GPU of its own for chat. The container must use host networking (or Ollama must listen on an address the container can reach) because Ollama binds to the loopback interface by default.
How many people can use Ollama at the same time?
By default one request per model is processed at a time and the rest are queued, up to 512. Raising OLLAMA_NUM_PARALLEL lets several answers generate together, and Ollama's documentation states that the memory needed scales with OLLAMA_NUM_PARALLEL times OLLAMA_CONTEXT_LENGTH. Raise it step by step and confirm with ollama ps that the model is still 100% on the GPU.
Which model should I pull on a 24 GB GPU and on a 96 GB GPU?
On 24 GB, models with a download of up to about 15 GB leave room for context: gpt-oss:20b (14 GB), qwen3:14b (9.3 GB) or gemma4:12b (8.0 GB). On 96 GB, llama3.3:70b (43 GB) and gpt-oss:120b (65 GB) fit with room for a long context. Sizes are from the Ollama library on 16 November 2026. The GPU dedicated server plans offer both card sizes.
Is Open WebUI open source and free for a company to use?
It is free to use, but since v0.6.6 its "Open WebUI License" is not an OSI open-source licence: the Open WebUI branding must stay unless the deployment has 50 or fewer users in a 30-day period, you have written permission or you hold an enterprise licence. Internal use with the branding intact needs no payment. Versions up to v0.6.5 are BSD-3-Clause.
Do prompts leave the server with this setup?
Not for inference. Ollama runs the model locally and Open WebUI stores chats in its own Docker volume on the server. Features you switch on yourself, such as web search or an external API connection, do send data out, so review the admin settings. If the data center location matters for your compliance work, ask sales which data center a server will be delivered in before ordering.
An Ollama server with 24 GB or 96 GB of VRAM
MassiveGRID GPU dedicated servers are bare metal with root access: an RTX PRO 4000 with 24 GB for $449.99 per month or an RTX PRO 6000 with 96 GB from $1,656.15 per month, each with a 1 Gbps port, unlimited traffic and a dedicated IPv4 address, hosted in an ISO 27001 certified data center. The server arrives with a clean OS; this guide takes it from there.
See Ollama HostingFurther Reading
- Ollama documentation: Linux install, systemd service and customising
- Ollama FAQ: bind address, keep-alive, concurrency settings, multiple GPUs
- Ollama documentation: context length defaults by VRAM
- Open WebUI quick start: Docker commands, WEBUI_SECRET_KEY, first admin account
- Open WebUI troubleshooting: connecting to Ollama with host networking
- Open WebUI License (v0.6.6 and later)
- Caddy: reverse proxy quick start and automatic HTTPS
- Ollama model library (tag sizes, context windows and licences)