vLLM: High-Throughput LLM Serving

By Khimananda Oli | Last reviewed: August 2026

Serving large language models at scale often hits a hard wall where GPU memory fragmentation throttles throughput long before compute saturation. vLLM: High-Throughput LLM Serving solves this bottleneck through PagedAttention and continuous batching, transforming how inference engines manage KV cache memory. If you are currently struggling with low tokens-per-second or excessive queue times, understanding these mechanisms is the first step toward efficient production deployment. For teams evaluating their broader infrastructure strategy, comparing GPU rental versus ownership costs provides essential context before optimizing the serving layer.

How does PagedAttention enable vLLM: High-Throughput LLM Serving?

The primary bottleneck in LLM inference is not raw compute but memory bandwidth and capacity. During autoregressive decoding, the model generates one token at a time, requiring the Key-Value (KV) cache to grow linearly with sequence length. Traditional serving frameworks pre-allocate contiguous memory for the maximum possible context window of every request. If a model supports 32k tokens but a specific prompt only uses 500, the remaining reserved space sits idle yet occupied, preventing other requests from using that VRAM. This internal fragmentation typically wastes 60–80% of GPU memory in production traffic mixes.

PagedAttention adapts the concept of virtual memory paging from operating systems to the KV cache. Instead of demanding contiguous memory, it partitions the KV cache into fixed-size blocks (typically 16 or 32 tokens). These blocks can be stored non-contiguously in physical GPU memory. A block table maps the logical token positions to their physical memory addresses. When a new token is generated, vLLM allocates only the specific block needed. If a sequence ends early, only its actual blocks are consumed; the rest remains available for other sequences.

Memory efficiency gains in practice

In my deployments serving Llama-3-70B on A100 80GB GPUs, switching from a static-allocation backend to vLLM consistently increased effective batch sizes by 2x to 4x without changing hardware. The elimination of fragmentation means you can serve longer average contexts or handle higher concurrency within the same VRAM envelope. This directly translates to lower cost-per-token, which is the metric that matters for production LLM cost optimization. Crucially, PagedAttention also enables efficient sharing of KV cache across parallel sampling or beam search, where multiple outputs share the same prompt prefix. Only one copy of the prefix blocks exists in memory, with reference counting managing lifecycle.

How do you configure continuous batching for maximum throughput?

Continuous batching (also called iteration-level scheduling) is the second pillar of vLLM: High-Throughput LLM Serving. Unlike static batching, which waits for all requests in a batch to finish before starting the next one, continuous batching adds new requests to the running batch at every single forward pass. As soon as one sequence finishes generating its stop token, it is evicted and replaced immediately by the next queued request. This keeps GPU utilization near 100% even when output lengths vary wildly.

1. Set max_num_seqs appropriately: This parameter caps concurrent sequences. Start with a value derived from your VRAM divided by the average KV cache size per sequence. On an A100 serving a 7B model, 256–512 is often sustainable; for 70B models, 32–64 may be the limit. Monitor GPU memory usage during load testing.
2. Tune max_model_len: Never set this higher than your actual use case requires. Each unit of max_model_len increases the block table overhead. If your p99 output is 2k tokens, setting max_model_len to 32k wastes metadata memory even with paging.
3. Enable chunked prefill: For mixed workloads with long prompts, enable --enable-chunked-prefill. This breaks large prefills into chunks interleaved with decode steps, preventing latency spikes for ongoing generations when a new long-context request arrives.
4. Configure scheduler policy: Use the default FCFS policy for general serving. Switch to priority-based scheduling only if you have distinct SLA tiers and understand the starvation risks.

# Production-ready vLLM launch command for Llama-3-8B-Instruct
python -m vllm.entrypoints.openai.api_server \
    --model meta-llama/Meta-Llama-3-8B-Instruct \
    --tensor-parallel-size 1 \
    --max-model-len 8192 \
    --max-num-seqs 256 \
    --gpu-memory-utilization 0.92 \
    --enable-chunked-prefill \
    --disable-log-requests \
    --port 8000

A common mistake is leaving gpu-memory-utilization at the default 0.9. In stable environments with no other GPU processes, raising this to 0.92–0.95 safely captures additional blocks for KV cache. However, exceeding 0.95 risks OOM crashes during transient CUDA allocations. Always validate with sustained load tests before committing to higher values.

How does vLLM compare to TGI and Ollama for production serving?

Choosing the right inference engine depends on your operational constraints. While Ollama excels for local development, production serving demands different trade-offs. The table below reflects benchmarks and operational experience from 2026 deployments on NVIDIA H100 and A100 hardware.

In practice, vLLM wins for general-purpose high-throughput serving where maximizing tokens-per-dollar is the objective. TGI can edge ahead in pure latency-sensitive scenarios with smaller batches due to its Rust runtime and aggressive kernel optimizations. Ollama should remain in your development and CI toolkit but rarely belongs behind a production load balancer handling thousands of RPM. For teams building RAG systems, pairing vLLM with proper vector database infrastructure ensures the retrieval layer doesn't become the new bottleneck.

How do you deploy vLLM on Kubernetes with autoscaling?

Running vLLM in Kubernetes requires treating GPU memory as the primary scaling signal, not CPU or generic memory. Standard HPA metrics fail because GPU VRAM pressure doesn't correlate with system RAM usage. Configure your deployment with explicit resource requests matching the GPU type, and use the vLLM Prometheus endpoint for custom metrics.

# vLLM Kubernetes Deployment (excerpt)
apiVersion: apps/v1
kind: Deployment
metadata:
  name: vllm-llama3-8b
spec:
  replicas: 2
  selector:
    matchLabels:
      app: vllm-llama3-8b
  template:
    spec:
      containers:
      - name: vllm
        image: vllm/vllm-openai:v0.8.5
        args:
          - "--model"
          - "meta-llama/Meta-Llama-3-8B-Instruct"
          - "--max-model-len"
          - "8192"
          - "--max-num-seqs"
          - "256"
          - "--gpu-memory-utilization"
          - "0.92"
        resources:
          limits:
            nvidia.com/gpu: 1
          requests:
            nvidia.com/gpu: 1
        ports:
        - containerPort: 8000
        readinessProbe:
          httpGet:
            path: /health
            port: 8000
          initialDelaySeconds: 60
          periodSeconds: 10

Autoscaling on queue depth

Use KEDA or Prometheus Adapter to scale based on vllm:num_requests_waiting or vllm:avg_generation_throughput. Queue depth is a leading indicator; GPU utilization is lagging. Set scale-up thresholds conservatively (e.g., queue > 10 for 30 seconds) and scale-down thresholds with longer cooldowns (5+ minutes) to avoid thrashing during traffic valleys. Cold starts for LLM containers are expensive—model loading takes 30–90 seconds—so maintain a minimum replica count of 1–2 even during low traffic. For teams managing multiple clusters or hybrid environments, integrating vLLM metrics into your existing Prometheus and Grafana monitoring stack provides unified visibility across inference and infrastructure layers.

What are the critical tuning parameters for vLLM performance?

Beyond the foundational flags, several advanced parameters determine whether your deployment achieves theoretical peak or stalls at 40% utilization. Profile your specific workload before applying these; blind tuning causes regressions.

• --tensor-parallel-size: Must equal the number of GPUs per node for tensor parallelism. For multi-node, combine with pipeline parallelism. Mismatched values cause silent fallback to slower execution paths.
• --swap-space: Allocates CPU RAM as overflow for KV cache blocks. Set to 4–8 GB per GPU for bursty workloads. Excessive swap degrades latency; insufficient swap causes request rejection under spike.
• --enforce-eager: Disables CUDA graph capture. Use only for debugging or models incompatible with graph capture. Production should always use CUDA graphs for 10–20% throughput gain.
• --prefix-caching: Enables automatic reuse of KV blocks for shared prefixes. Critical for system-prompt-heavy applications like chatbots or RAG. Disabled by default in some versions; explicitly enable.
• --speculative-model: Configures speculative decoding with a smaller draft model. Can improve latency-bound workloads by 1.5–2x but increases memory overhead. Benchmark thoroughly; benefits are workload-dependent.

Monitor vllm:gpu_cache_usage_perc closely. Sustained values above 90% indicate imminent queuing; values below 50% suggest over-provisioning or overly conservative max_num_seqs. Pair this with vllm:avg_prompt_throughput and vllm:avg_generation_throughput to distinguish prefill-bound from decode-bound regimes. Adjust chunked prefill size and batch limits accordingly.

Optimizing Your vLLM Deployment for Production

vLLM: High-Throughput LLM Serving represents the current standard for cost-efficient production inference, but its advantages only materialize through deliberate configuration and monitoring. Start with conservative parameters, instrument everything via Prometheus, and iterate based on real traffic patterns rather than synthetic benchmarks. The gap between a default vLLM deployment and a tuned one is often 3x in effective throughput. If your team needs assistance architecting GPU infrastructure, configuring autoscaling policies, or integrating inference metrics into existing observability platforms, reach out to discuss your specific deployment requirements.