Run LLMs Locally with Ollama and vLLM

August 10, 2026 Khimananda Oli 7 min read Virtualization

By Khimananda Oli | Last reviewed: August 2026

Data sovereignty and API costs are the two primary drivers pushing engineering teams to run LLMs locally with Ollama and vLLM instead of relying solely on managed cloud providers. While cloud APIs offer convenience, they introduce latency, recurring token fees, and compliance risks that many regulated industries in Nepal and abroad cannot accept. This guide provides a practitioner’s blueprint for deploying self-hosted inference servers, helping you choose the right toolchain for development versus high-throughput production workloads.

Quick answer: To run LLMs locally with Ollama and vLLM, install Ollama for rapid prototyping and CLI-based testing, then deploy vLLM with PagedAttention for production-grade throughput. Ollama prioritizes ease of use and model management, while vLLM optimizes GPU memory for concurrent serving.

Local LLM architecture: Ollama for development and vLLM for production serving

How do you run LLMs locally with Ollama and vLLM for different workflows?

The decision to run LLMs locally with Ollama and vLLM is rarely an either/or choice; mature teams typically use both tools at different stages of the AI lifecycle. Understanding this distinction prevents the common mistake of trying to force a development tool into production or over-engineering a local testing environment. Before diving into installation, review our primer on self-hosting an LLM: options, costs, and GPU requirements to ensure your hardware baseline matches your target model size.

Ollama: The Developer Experience Layer

Ollama abstracts away the complexity of model quantization, dependency management, and runtime configuration. It uses GGUF-formatted models, which are heavily quantized (typically Q4_K_M or Q5_K_M) to run efficiently on consumer hardware, Apple Silicon, and laptops without dedicated NVIDIA GPUs. In practice, Ollama serves as your validation sandbox. You can pull, test, and iterate on prompts in seconds. Its REST API is compatible with many open-source libraries, making it ideal for building proof-of-concept RAG systems or integrating AI into local DevOps scripts.

vLLM: The Production Serving Engine

vLLM is designed exclusively for high-throughput serving on NVIDIA GPUs (and increasingly AMD/Intel accelerators). It utilizes PagedAttention, a memory management algorithm that eliminates KV-cache fragmentation, allowing you to serve significantly more concurrent requests per GPU than standard HuggingFace Transformers. Unlike Ollama, vLLM typically loads full-precision or AWQ/GPTQ quantized weights directly from HuggingFace. It exposes a fully compliant OpenAI-compatible API server, meaning your application code does not need to change when migrating from a managed provider to self-hosted infrastructure.

What are the hardware requirements to run LLMs locally with Ollama and vLLM?

Hardware dictates which models you can load and what throughput you can sustain. A frequent failure mode in 2026 is underestimating VRAM overhead for the KV cache during concurrent serving. While model weights are static, the KV cache grows linearly with context length and batch size.

Model Tier	VRAM (Weights Only)	Recommended GPU	Best Tool	Notes
7B–8B (Q4_K_M)	~5 GB	RTX 4060 / M2 Pro	Ollama	Ideal for local dev and testing
7B–8B (FP16/AWQ)	~16 GB	RTX 4090 / L4	vLLM	Entry-level production serving
70B (Q4_K_M)	~40 GB	A6000 / 2x RTX 3090	Ollama	Slow but functional for eval
70B (AWQ/GPTQ)	~48 GB	A100 80GB / H100	vLLM	Standard for enterprise RAG
405B+ (Quantized)	~120 GB+	Multi-GPU Cluster	vLLM	Requires tensor parallelism

For teams in Nepal or regions with limited access to latest-gen hardware, older RTX 3090s (24GB VRAM) remain excellent value for running 7B–14B production models with vLLM. Always reserve at least 20% of VRAM headroom beyond the model weight size to accommodate activation memory and KV cache during peak concurrency.

How do you configure vLLM for maximum throughput and stability?

Running vLLM with default settings leaves significant performance on the table. Proper configuration requires tuning memory allocation, tensor parallelism, and request scheduling based on your specific workload characteristics.

Set GPU Memory Utilization Explicitly: Default utilization is often conservative. For dedicated inference servers, increase --gpu-memory-utilization 0.95 to maximize KV cache capacity.
Enable Chunked Prefill: For mixed workloads with varying prompt lengths, use --enable-chunked-prefill. This prevents long-context requests from blocking shorter ones, improving overall tail latency.
Configure Tensor Parallelism: If using multiple GPUs, set --tensor-parallel-size N matching your physical GPU count. Do not use pipeline parallelism unless crossing node boundaries; tensor parallelism has lower communication overhead within a single node.
Optimize Max Model Length: Set --max-model-len to the actual maximum context your application requires, not the model's theoretical limit. This pre-allocates appropriate KV cache blocks and prevents out-of-memory errors during traffic spikes.
Use Quantized Checkpoints: Prefer AWQ or GPTQ 4-bit checkpoints over FP16 for serving. They offer near-identical quality with 3–4x higher throughput on modern GPUs.

# Production vLLM launch command for Llama-3-70B-Instruct-AWQ
python -m vllm.entrypoints.openai.api_server \
    --model casperhansen/llama-3-70b-instruct-awq \
    --tensor-parallel-size 2 \
    --gpu-memory-utilization 0.92 \
    --max-model-len 8192 \
    --enable-chunked-prefill \
    --port 8000 \
    --host 0.0.0.0

PagedAttention eliminates KV-cache fragmentation by mapping logical tokens to non-contiguous physical GPU memory blocks

How do you integrate local LLMs into existing DevOps and CI pipelines?

Self-hosted models become valuable when integrated into automated workflows rather than sitting idle as chat endpoints. When you run LLMs locally with Ollama and vLLM, you gain deterministic, low-latency access that enables real-time automation. For practical integration patterns, see our guide on running local LLMs with Ollama for DevOps workflows.

CI Code Review Bots: Deploy a lightweight 7B model via Ollama as a GitHub Actions service container. It can perform initial security scans and style checks before human review, reducing reviewer fatigue. Because it runs locally within the runner, no secrets leave your CI environment.
Log Anomaly Detection: Stream structured logs to a local vLLM instance fine-tuned on incident postmortems. Pair this with AI-powered log analysis techniques to surface root causes faster than keyword grep.
Terraform Plan Summarization: Pipe terraform plan -json output to a local model to generate human-readable change summaries for PR descriptions. This catches unintended resource deletions before apply.
Documentation Generation: Use batch inference with vLLM to regenerate API docs from OpenAPI specs nightly. Local execution avoids per-token costs that make large-scale doc regeneration prohibitive with cloud APIs.

When should you choose Ollama over vLLM for production workloads?

Despite vLLM's performance advantages, Ollama remains the better choice for specific production scenarios where raw throughput is secondary to operational simplicity or hardware constraints.

CPU-Only Environments: If your edge deployment lacks NVIDIA GPUs (e.g., Raspberry Pi clusters, older VMs), Ollama's GGUF backend is optimized for CPU inference. vLLM requires CUDA or ROCm.
Rapid Model Swapping: For multi-tenant platforms where users request different models dynamically, Ollama's hot-loading and automatic unloading manage memory more gracefully than vLLM's static allocation.
Single-User Desktop Apps: Embedded applications like IDE assistants or personal knowledge bases benefit from Ollama's minimal footprint and background daemon architecture.
Evaluation and Benchmarking: When testing dozens of model variants weekly, Ollama's pull-and-run workflow eliminates the overhead of converting weights and configuring serving parameters.

Conversely, any workload exceeding 10 concurrent requests, requiring sub-second time-to-first-token, or serving models larger than 14B parameters should default to vLLM. The operational complexity of managing Python dependencies and CUDA versions is offset by orders-of-magnitude better resource utilization.

Decision framework: select Ollama or vLLM based on GPU availability and concurrency requirements

Deploy Secure Local Inference That Scales

Choosing to run LLMs locally with Ollama and vLLM gives you control over data residency, cost predictability, and latency that cloud APIs cannot match. Start with Ollama to validate your models and prompts, then graduate to vLLM when throughput demands justify the operational investment. Remember that self-hosting shifts responsibility for security patching, GPU driver updates, and monitoring onto your team — treat your inference server with the same rigor as any production database. If you need help designing a compliant, audit-ready local AI infrastructure or optimizing your current setup, reach out to discuss your deployment.

Frequently Asked Questions

Ollama prioritizes ease of use with simple CLI commands and model management for individual developers. vLLM focuses on high-throughput production serving using PagedAttention optimization, making it better suited for concurrent API requests and enterprise deployments requiring maximum tokens per second on dedicated GPU hardware.

Yes, but you must configure VRAM limits to prevent out-of-memory errors. Set CUDA_VISIBLE_DEVICES or specific memory fractions in vLLM while restricting Ollama's context window. Monitor usage with nvidia-smi to ensure combined allocation stays within your physical GPU memory constraints during 2026 workloads.

vLLM significantly outperforms Ollama for concurrent requests due to continuous batching and PagedAttention memory management. Benchmarks in 2026 show vLLM handling twenty to fifty simultaneous streams efficiently, whereas Ollama processes requests sequentially or with limited parallelism, causing latency spikes under heavy production load.

Yes.

A 70B model at Q4 quantization requires approximately forty gigabytes of VRAM for comfortable inference with moderate context. Full precision demands over one hundred gigabytes. Use dual RTX 4090s or a single A100/A6000 Ada for viable local deployment without excessive CPU offloading bottlenecks in 2026.

No.

Ollama natively serves an OpenAI-compatible endpoint at localhost colon 11434 slash v1. Configure your application base URL to this address without additional proxies. Authentication is disabled by default, so place Nginx or Caddy in front for TLS termination and API key validation before exposing to any network.

Insufficient batch size configuration often leaves GPU compute idle. Increase max_num_seqs and gpu_memory_utilization flags during startup. Also verify tensor parallelism matches your GPU count exactly. Profiling with nsight-systems identifies whether bottlenecks stem from memory bandwidth, kernel launches, or suboptimal scheduling parameters.

Neither tool supports training or fine-tuning natively. Use Unsloth, Axolotl, or LLaMA-Factory for efficient LoRA adaptation, then export weights to GGUF for Ollama or HuggingFace format for vLLM. Both tools are strictly inference engines designed for serving pre-trained or adapted models locally.

Q4_K_M and Q5_K_M retain near-lossless quality for most tasks while halving VRAM requirements compared to FP16. Lower quantizations like Q2_K degrade reasoning and instruction following noticeably. Always benchmark perplexity on your specific domain data before deploying heavily compressed models in production environments during 2026.

Default configurations lack authentication, rate limiting, and input sanitization. Attackers can extract sensitive context, exhaust resources via prompt injection, or pivot to internal services. Always enforce API keys, implement request quotas, validate inputs, and isolate inference containers behind reverse proxies with strict firewall rules.

Pull new versions alongside existing ones using ollama pull, then update your application routing gradually. Ollama keeps old models cached until explicitly removed. For zero-downtime swaps in production, use blue-green deployment patterns with separate Ollama instances behind a load balancer during 2026 maintenance windows.

Yes, vLLM supports speculative decoding using a smaller draft model to propose tokens verified by the target model. Enable via the speculative_model flag during server startup. This technique accelerates generation two to three times for structured outputs like code or JSON without sacrificing accuracy.

The requested context length exceeds available VRAM after model weights load. Reduce num_ctx parameter or switch to higher quantization. Check ollama ps to see current allocations. System RAM offloading works but drastically slows inference; upgrade GPU memory or use smaller models for reliable local operation.

Ollama offers superior LangChain integration through dedicated ChatOllama and OllamaEmbeddings classes with minimal configuration. vLLM requires generic OpenAI-compatible wrappers since it lacks native LangChain bindings. For rapid prototyping and local agent development in 2026, Ollama reduces boilerplate significantly compared to configuring vLLM endpoints manually.