Self-Host LLMs with Ollama and Open WebUI

August 14, 2026 Khimananda Oli 7 min read Virtualization

By Khimananda Oli | Last reviewed: August 2026

Running proprietary AI models through public APIs creates data residency risks, unpredictable costs, and latency issues that many teams in Nepal and abroad can no longer ignore. When you self-host LLMs with Ollama and Open WebUI, you regain full control over your inference stack while keeping sensitive data entirely within your own infrastructure. This guide walks you through a production-grade deployment using Docker, covering GPU acceleration, persistent storage, and security hardening based on real-world implementations I have managed throughout 2026.

Quick answer: To self-host LLMs with Ollama and Open WebUI, deploy both services via Docker Compose with GPU passthrough enabled. Ollama serves as the local inference backend managing model weights, while Open WebUI provides the chat interface and RAG pipeline. Bind volumes for persistence and restrict network access to keep your private AI stack secure and compliant.

High-level architecture to self-host LLMs with Ollama and Open WebUI showing container boundaries and GPU passthrough

How do you install Ollama and Open WebUI with Docker Compose?

The most reliable way to self-host LLMs with Ollama and Open WebUI in 2026 is through Docker Compose. This approach ensures reproducible deployments, simplifies updates, and makes it straightforward to apply the security hardening principles covered in my Ubuntu security hardening guide. Before starting, verify that your host has the NVIDIA Container Toolkit installed if you plan to use GPU acceleration, as the runtime will fail silently otherwise.

Create the compose file

Create a file named docker-compose.yml in your project directory. This configuration mounts named volumes for both model weights and application state, preventing data loss during container restarts or upgrades.

version: "3.9"
services:
  ollama:
    image: ollama/ollama:latest
    container_name: ollama
    restart: unless-stopped
    ports:
      - "127.0.0.1:11434:11434"
    volumes:
      - ollama_models:/root/.ollama
    deploy:
      resources:
        reservations:
          devices:
            - driver: nvidia
              count: all
              capabilities: [gpu]
    environment:
      - OLLAMA_KEEP_ALIVE=24h
      - OLLAMA_NUM_PARALLEL=4

  open-webui:
    image: ghcr.io/open-webui/open-webui:main
    container_name: open-webui
    restart: unless-stopped
    ports:
      - "3000:8080"
    volumes:
      - open_webui_data:/app/backend/data
    environment:
      - OLLAMA_BASE_URL=http://ollama:11434
      - WEBUI_AUTH=true
      - ENABLE_SIGNUP=false
    depends_on:
      - ollama

volumes:
  ollama_models:
  open_webui_data:

Start the stack and pull a model

Run docker compose up -d to start both services in detached mode.
Verify GPU detection with docker exec ollama nvidia-smi.
Pull your first model: docker exec ollama ollama pull llama3.1:8b.
Access the interface at http://localhost:3000 and create an admin account.

A common mistake is binding Ollama’s port to 0.0.0.0. Always bind to 127.0.0.1 unless you have a reverse proxy with authentication in front. Exposing the raw Ollama API to the internet allows anyone to execute arbitrary prompts and consume your compute resources without restriction.

What hardware do you need to self-host LLMs effectively?

Hardware requirements depend entirely on the model size and quantization level you intend to run. When you self-host LLMs with Ollama and Open WebUI, VRAM is the primary bottleneck. System RAM matters only when models spill over from GPU memory, which drastically reduces token generation speed. For teams evaluating whether to rent vs buy GPUs for AI workloads, understanding these thresholds prevents costly over-provisioning.

Model Size	Quantization	Min VRAM	Recommended GPU	Use Case
7B–8B	Q4_K_M	6 GB	RTX 4060 / T4	Chat, summarization, dev assistance
13B–14B	Q4_K_M	10 GB	RTX 4070 Ti / A10G	RAG, code completion, analysis
30B–34B	Q4_K_M	20 GB	RTX 4090 / A10	Complex reasoning, multi-step tasks
70B	Q4_K_M	40 GB	A100 40GB / 2× RTX 4090	Production-grade general intelligence

If you are running on CPU-only hardware, expect 2–5 tokens per second for 8B models. That is usable for batch processing but frustrating for interactive chat. Apple Silicon Macs with unified memory perform surprisingly well due to high bandwidth, though they lack CUDA ecosystem compatibility. For Nepali organizations dealing with import restrictions, a single RTX 4090 workstation often delivers better price-performance than cloud GPU rentals for sustained internal workloads.

Request flow when you self-host LLMs with Ollama and Open WebUI demonstrating RAG document retrieval before inference

How do you configure RAG and persistent storage in Open WebUI?

Retrieval-Augmented Generation transforms a generic model into a domain-specific assistant. Open WebUI includes a built-in vector store powered by ChromaDB or Elasticsearch, eliminating the need for separate infrastructure like external vector databases for most team-scale deployments. Documents uploaded through the UI are chunked, embedded using a local sentence-transformer model, and stored in the mounted volume.

Optimize chunking and embedding

Default settings work for quick tests but produce poor retrieval quality for technical documentation. Adjust these parameters in the Admin Panel → RAG Settings:

Chunk Size: Set to 512 tokens for code repositories, 1024 for policy documents.
Chunk Overlap: Use 10–15% of chunk size to preserve context across boundaries.
Embedding Model: Use nomic-embed-text or bge-m3 instead of the default. Pull via Ollama: ollama pull nomic-embed-text.
Top-K: Start with 5 results. Increase only if recall is insufficient; higher values add noise.

Store embeddings on fast NVMe storage. Vector search latency directly impacts perceived responsiveness. If you are running on a VPS with network-attached storage, consider dedicating a local SSD partition to the open_webui_data volume. Back up this volume regularly using strategies from my server backup guide, as re-indexing large document collections takes hours.

How do you secure a self-hosted LLM stack for production?

Security is where most self-hosted AI deployments fail. Treating your local LLM stack like a toy leads to data leaks and compliance violations. When you self-host LLMs with Ollama and Open WebUI for business use, apply the same rigor you would to any production service handling sensitive information.

Network and access controls

Reverse Proxy: Place Nginx or Caddy in front of Open WebUI. Terminate TLS there. Never expose port 8080 directly.
Authentication: Enable WEBUI_AUTH=true and disable public signup after creating admin accounts. Integrate with OIDC/LDAP for team environments.
Firewall Rules: Allow inbound traffic only on ports 443 (HTTPS). Block all direct access to 11434 and 8080 from external interfaces.
Rate Limiting: Configure rate limits at the reverse proxy layer. A single user running infinite loops can saturate GPU memory and deny service to others.

Data governance and compliance

For Nepali fintech or healthcare organizations, data residency is non-negotiable. Self-hosting solves this inherently, but you must still implement audit trails. Open WebUI logs all conversations to its SQLite database. Export these logs periodically and integrate with centralized logging systems like those described in my structured logging best practices article. Encrypt the Docker volumes at rest using LUKS or filesystem-level encryption. Regularly review which models are pulled and remove unused ones to reduce attack surface and disk consumption.

Trade-off comparison when deciding to self-host LLMs with Ollama and Open WebUI versus using cloud APIs

When should you choose self-hosting over cloud LLM APIs?

Self-hosting is not universally superior. It trades operational simplicity for control and privacy. Choose to self-host LLMs with Ollama and Open WebUI when data cannot leave your jurisdiction, when monthly API spend exceeds hardware amortization costs, or when you need offline operation. Cloud APIs remain better for bursty workloads, cutting-edge frontier models, or teams without DevOps capacity. For a deeper analysis of this decision framework, see my guide on building vs buying LLM features.

In practice, many organizations adopt a hybrid approach. Run routine internal tasks on self-hosted 8B–14B models for speed and privacy. Route complex reasoning or customer-facing outputs to cloud APIs with proper PII redaction. Open WebUI supports configuring multiple Ollama endpoints and external API providers simultaneously, making this split transparent to end users.

Getting Started with Your Private AI Stack

Deploying a private inference stack removes vendor dependency and keeps sensitive data under your control. Start with an 8B quantized model on available hardware to validate your workflow before investing in dedicated GPUs. Monitor GPU utilization and token throughput to right-size your infrastructure. If you need help designing a compliant, production-ready AI infrastructure for your team, reach out to discuss your requirements.

Frequently Asked Questions

You need at least 16GB RAM for 7B models and an NVIDIA GPU with 8GB VRAM. CPU-only setups work but are slow. For production in 2026, allocate 32GB RAM and a RTX 4090 or Apple M3 Max for acceptable token generation speeds.

Create a docker-compose.yml defining both services with shared volumes. Map port 11434 for Ollama and 3000 for Open WebUI. Set the OLLAMA_BASE_URL environment variable in the WebUI container to http://ollama:11434. Run docker compose up -d to start both containers simultaneously.

No, never expose Ollama or Open WebUI directly without authentication. Use a reverse proxy like Caddy or Nginx with TLS termination and basic auth or OAuth2-proxy. Enable API keys in Open WebUI settings and restrict network access via firewall rules to prevent unauthorized model execution or data leakage.

Yes, Ollama supports AMD ROCm on Linux. Install the rocm-hip-runtime package and set HSA_OVERRIDE_GFX_VERSION for unsupported cards. Performance varies by architecture; RDNA3 chips perform best. Verify detection with ollama run llama3 then check gpu info in the server logs.

Initial hardware costs range from $800 to $3000 depending on GPU choice. Monthly electricity adds $20-$50 under load. Break-even versus OpenAI API occurs around six months for teams processing over two million tokens daily. Long-term savings justify upfront investment for high-volume internal applications.

Llama 3.2, Mistral-Nemo, and Qwen2.5 offer the best quality-to-resource ratio. Use quantized GGUF variants like Q4_K_M for consumer hardware. Pull specific tags via ollama pull llama3.2:8b-q4_K_M rather than latest to ensure consistent performance and predictable memory usage across deployments.

Check if the model fits entirely in VRAM using nvidia-smi during inference. Partial offloading to system RAM causes significant slowdowns. Reduce context window size or switch to smaller quantization. Ensure CUDA drivers match your Ollama version and that no other processes compete for GPU resources.

Pull new model versions alongside existing ones using ollama pull. Update Open WebUI model selection after verification. Delete old tags only after confirming stability. Since Ollama serves multiple models concurrently, you can test new versions while production traffic continues using the previous stable tag.

Yes, configure multiple OLLAMA_BASE_URL endpoints in Open WebUI admin settings for load distribution or model segregation. This enables routing specific requests to dedicated hardware. Useful for separating embedding models from chat completion workloads across different machines in your self-hosted infrastructure.

Mount a named Docker volume to /app/backend/data in your compose file. This stores SQLite databases, uploaded files, and user configurations outside the container. Regular backups of this volume protect against accidental deletion. Avoid bind mounts on network storage due to SQLite locking issues.

Yes, models like Llama 3.2 and Mistral support native function calling through Ollama. Define tools in JSON schema format when making API calls. Open WebUI provides a visual interface for configuring custom tools and RAG pipelines that integrate with local document stores and external APIs.

Ollama offers better CLI automation, Docker integration, and API compatibility for production deployments. LM Studio provides superior GUI model exploration and testing. Choose Ollama for server environments and team access via Open WebUI. Use LM Studio for individual experimentation before committing to specific model selections.

Upload documents through the workspace interface and select an embedding model like nomic-embed-text. Open WebUI automatically chunks content and stores vectors in ChromaDB. Configure retrieval parameters in admin settings. All processing stays local, ensuring sensitive documents never leave your self-hosted infrastructure during indexing or querying.

Yes, create user groups in admin panel and assign model permissions per group. Developers might access raw base models while end-users see only fine-tuned variants. Combine with API key rotation and audit logging to maintain compliance. Permissions apply to both chat interface and programmatic API access.

Expose Prometheus metrics from Ollama using the built-in endpoint at /metrics. Deploy Grafana dashboards tracking GPU utilization, queue depth, and token throughput. Add container exporters for memory and CPU monitoring. Set alerts for sustained high utilization indicating need for scaling or model optimization in your deployment.