Serve LLMs in Production: Throughput and Latency

Q: What is the difference between throughput and latency in LLM serving?

Throughput measures tokens generated per second across all concurrent requests, while latency tracks time-to-first-token or total response time for individual users. Optimizing one often impacts the other, requiring balanced configuration based on your specific production workload requirements and user experience targets.

Q: How does continuous batching improve LLM serving throughput?

Continuous batching dynamically adds new requests to running batches instead of waiting for completion, maximizing GPU utilization. This technique significantly increases tokens per second compared to static batching by eliminating idle compute cycles during variable-length sequence generation in high-concurrency production environments.

Q: Which inference engine offers the best latency for Llama 3 models?

vLLM with PagedAttention currently delivers lowest latency for Llama 3 variants through efficient KV cache management. TensorRT-LLM provides comparable performance with NVIDIA-specific optimizations, while SGLang excels at structured generation workloads requiring constrained output formats and complex prompting patterns.

Q: Does quantization always reduce LLM serving latency?

Not necessarily. While INT4 or INT8 quantization reduces memory bandwidth pressure and can accelerate throughput, some kernels introduce computational overhead that increases per-token latency. Benchmark your specific model and hardware combination using tools like lmperf before deploying quantized weights to production.

Q: What GPU memory is needed to serve 70B parameter models?

A single 70B FP16 model requires approximately 140GB VRAM, necessitating multi-GPU tensor parallelism. AWQ 4-bit quantization reduces this to roughly 35GB, fitting on one A100 80GB or H100. Always reserve 10-20% headroom for KV cache during concurrent request handling.

Q: How do I measure time-to-first-token accurately in production?

Instrument your inference server to log timestamp deltas between request receipt and first output token emission. Use percentile metrics like p50 and p99 rather than averages, as tail latency reveals queuing bottlenecks. Tools like locust or vegeta help generate realistic load patterns for measurement.

Q: Can speculative decoding reduce LLM response latency?

Yes, speculative decoding uses a smaller draft model to propose tokens verified in parallel by the larger target model, reducing sequential decoding steps. Expect 1.5x to 3x latency improvements for long generations, though benefits diminish for short outputs or highly creative tasks with low acceptance rates.

Q: What causes sudden latency spikes in LLM serving?

Common culprits include KV cache eviction under memory pressure, garbage collection pauses, thermal throttling, or batch size miscalculation during traffic surges. Monitor GPU memory fragmentation, CPU scheduling delays, and request queue depth to diagnose intermittent performance degradation in production serving infrastructure.

Q: Is prefix caching worth implementing for chat applications?

Absolutely. Prefix caching reuses computed KV states for shared system prompts or conversation history, eliminating redundant computation. For chat workloads with lengthy repeated contexts, this reduces time-to-first-token by 40-70% and dramatically improves throughput during peak concurrent user sessions.

Q: How does tensor parallelism affect serving latency versus throughput?

Tensor parallelism splits model weights across GPUs, reducing per-device memory but introducing communication overhead. This lowers latency for large models that cannot fit single-GPU memory, yet may decrease overall throughput due to synchronization costs. Profile interconnect bandwidth before scaling beyond two GPUs.

August 10, 2026 Khimananda Oli 9 min read Virtualization

By Khimananda Oli | Last reviewed: August 2026

When you serve LLMs in production: throughput and latency are opposing forces that determine your unit economics and user experience. Most teams fail because they optimize for one metric in isolation, either burning cash on idle GPUs or delivering sluggish responses during traffic spikes. This guide covers the architectural patterns, runtime configurations, and infrastructure decisions required to balance both metrics effectively in 2026.

Quick answer: To serve LLMs in production: throughput and latency require continuous batching runtimes like vLLM or SGLang, KV-cache-aware memory management (PagedAttention), and dynamic GPU autoscaling. Prioritize time-to-first-token for interactive apps and tokens-per-second for batch workloads, validating both via load testing before going live.

How do you architect a system to serve LLMs in production: throughput and latency trade-offs?

The fundamental challenge when you self-host an LLM is that transformer inference is memory-bandwidth bound during the decode phase but compute-bound during the prefill phase. A naive sequential processing approach leaves GPU tensor cores idle while waiting for HBM transfers. Modern serving architectures solve this through continuous batching and disaggregated prefill/decode stages.

High-level architecture to serve LLMs in production: throughput and latency depend on continuous batching schedulers distributing requests across GPU workers with shared KV cache management.

In practice, continuous batching allows new requests to join an ongoing generation step without waiting for the entire batch to finish. This dramatically improves GPU utilization compared to static batching. The scheduler must be aware of each sequence's KV cache size to avoid out-of-memory errors mid-generation. When designing this layer, consider these critical factors:

Prefill vs. Decode Disaggregation: For mixed workloads, separate prefill-heavy and decode-heavy requests onto different GPU pools. Prefill is compute-intensive; decode is memory-bandwidth-bound. Mixing them causes resource contention.
KV Cache Offloading: Implement tiered caching where active sequences stay in HBM, paused sequences move to host RAM, and completed prefixes persist to NVMe. This enables serving more concurrent users than GPU memory alone permits.
Request Priority Queues: Interactive chat requests need low TTFT (time-to-first-token); batch summarization jobs need high TPS (tokens-per-second). Route them through separate queues with different scheduling policies.
Speculative Decoding: Use a smaller draft model to propose tokens that the larger target model verifies in parallel. This can improve decode throughput by 2-3x for predictable outputs without quality loss.

Which inference runtime delivers the best performance for production LLM serving?

The runtime you choose dictates your ceiling for optimization. In 2026, three runtimes dominate production deployments, each with distinct strengths depending on whether you prioritize raw throughput, latency consistency, or operational simplicity.

Runtime	Best For	Key Optimization	TTFT (p95)	TPS (Batch=32)	Production Maturity
vLLM	General-purpose serving	PagedAttention, continuous batching	~180ms	~2,400 tok/s	High (industry standard)
SGLang	Complex prompting, RAG pipelines	RadixAttention, prefix caching	~150ms	~2,800 tok/s	Medium-High
TensorRT-LLM	NVIDIA-only, max throughput	Fused kernels, INT4/FP8 quantization	~120ms	~3,500 tok/s	High (NVIDIA ecosystem)
TGI	HuggingFace-native workflows	Speculative decoding, watermarking	~220ms	~2,000 tok/s	Medium

For most teams learning how to deploy ML models in production, vLLM remains the safest starting point due to its broad hardware support and OpenAI-compatible API. However, if your workload involves heavy prefix sharing (system prompts, few-shot examples, RAG contexts), SGLang's RadixAttention automatically caches and reuses KV states across requests, reducing redundant computation by 40-60%.

Configuring vLLM for balanced throughput and latency

A common mistake is running vLLM with default settings. Production deployments require explicit tuning of memory allocation, batch sizes, and chunked prefill. Here is a battle-tested configuration for an A100 80GB serving Llama-3.1-70B-Instruct-AWQ:

<!-- vllm-launch.sh -->
python -m vllm.entrypoints.openai.api_server \
  --model casperhansen/llama-3.1-70b-instruct-awq \
  --tensor-parallel-size 2 \
  --gpu-memory-utilization 0.92 \
  --max-model-len 8192 \
  --enable-chunked-prefill \
  --max-num-batched-tokens 16384 \
  --max-num-seqs 256 \
  --scheduler-delay-factor 0.1 \
  --disable-log-requests \
  --port 8000

Key parameters explained:

--gpu-memory-utilization 0.92: Reserves 8% headroom for CUDA context fragmentation and activation peaks. Setting this to 0.95+ risks OOM during bursty traffic.
--enable-chunked-prefill: Breaks long prefills into chunks interleaved with decode steps. Prevents TTFT spikes when a single 4K-token request blocks the entire batch.
--max-num-seqs 256: Caps concurrent sequences. Higher values increase throughput but degrade per-request latency. Benchmark your specific SLA to find the sweet spot.
--scheduler-delay-factor 0.1: Introduces micro-delays to accumulate larger batches before scheduling. Trades ~10ms TTFT for 15-20% TPS gains under moderate load.

How does GPU memory management impact LLM serving throughput?

Memory is the bottleneck, not compute. Transformer attention scales quadratically with context length, and KV cache consumption grows linearly with both batch size and sequence length. Without intelligent memory management, you will either waste expensive HBM capacity or crash under load.

Static KV cache allocation wastes 30-40% of HBM on padding; PagedAttention uses virtual block mapping to achieve near-optimal memory utilization when serving LLMs in production.

PagedAttention treats KV cache like operating system virtual memory: fixed-size blocks (typically 16 or 32 tokens) are allocated on demand and mapped via a block table. This eliminates the need to reserve contiguous memory for maximum possible sequence length. In my experience deploying 70B-class models, PagedAttention consistently delivers 2.5-3x higher concurrent request capacity versus static allocation.

Quantization: the throughput multiplier

AWQ and GPTQ 4-bit quantization reduce memory footprint by ~75% with negligible quality degradation for instruction-tuned models. This directly translates to higher batch sizes and better memory bandwidth utilization. For Llama-3.1-70B, AWQ fits comfortably on 2×A100-80GB with room for 8K context at batch 256, while FP16 requires 4×A100s for equivalent concurrency.

# Verify AWQ model loads correctly and measure baseline
python -c "
from vllm import LLM, SamplingParams
llm = LLM('casperhansen/llama-3.1-70b-instruct-awq', 
          tensor_parallel_size=2, 
          gpu_memory_utilization=0.92)
outputs = llm.generate(['Explain Kubernetes pod scheduling'], 
                       SamplingParams(max_tokens=128))
print(f'Tokens generated: {len(outputs[0].token_ids)}')
"

How do you benchmark and monitor LLM serving performance in production?

You cannot optimize what you do not measure. Synthetic benchmarks mislead because they ignore real-world request distributions, network overhead, and cold-start effects. Build a benchmarking harness that mirrors your actual production traffic patterns.

Load testing with realistic request profiles

Use tools like genai-perf (NVIDIA) or locust-llm to generate load with configurable input/output token distributions. Always test three scenarios:

Steady-state: Constant QPS matching your p50 production load for 10+ minutes. Measures sustainable throughput.
Burst: 5x normal QPS for 60 seconds. Validates scheduler backpressure and queue behavior.
Mixed-length: Combine short (chat) and long (document analysis) requests. Exposes chunked prefill effectiveness.

Critical metrics to capture at p50, p95, and p99:

TTFT (Time to First Token): User-perceived latency. Target <300ms p95 for interactive apps.
ITL (Inter-Token Latency): Streaming smoothness. Target <50ms p95.
TPOT (Time Per Output Token): Decode efficiency. Directly correlates with GPU utilization.
Throughput (req/s and tok/s): Capacity planning metric. Track cost per million tokens.
Queue depth and wait time: Early warning signal for overload before latency degrades.

Integrate these metrics into your LLMOps monitoring stack with Prometheus exporters. Set alerts on p95 TTFT exceeding SLA thresholds and GPU memory utilization crossing 90% sustained. For teams implementing predictive autoscaling, feed historical TTFT and queue depth into your forecasting model to preemptively scale GPU nodes before user-facing degradation occurs.

Essential monitoring panels when you serve LLMs in production: throughput and latency visibility requires tracking TTFT percentiles, GPU memory pressure, queue depth, and cost efficiency in real time.

What infrastructure choices minimize cost while maintaining LLM serving SLAs?

Hardware selection determines your cost floor. For 70B-class models in 2026, the optimal configurations balance memory bandwidth, interconnect speed, and price-per-token:

A100 80GB (PCIe): Best value for most teams. ~$1.80/M output tokens at batch 128. Widely available, mature tooling.
H100 SXM: 2x throughput over A100 for same model. Justified only above 500 req/min sustained. ~$1.20/M tokens but higher minimum spend.
L40S / RTX 6000 Ada: Cost-effective for <30B models or quantized 70B. Lower HBM bandwidth limits max batch size.
Multi-GPU Interconnect: NVLink/NVSwitch matters enormously for tensor parallelism. PCIe TP adds 15-25% latency overhead per additional GPU. Avoid 4-way PCIe TP for latency-sensitive workloads.

For Nepal-based teams or those serving South Asian users, consider regional GPU cloud providers in Singapore or Mumbai to keep network latency under 100ms. Data residency requirements for Nepali fintech or government projects may necessitate on-premise deployments; in such cases, budget for 2×A100 nodes with NVLink and implement aggressive KV cache offloading to maximize utilization. Review LLM cost optimization strategies for detailed breakdowns of self-hosted versus API pricing at various traffic levels.

Autoscaling that respects warm-up time

GPU instances take 3-8 minutes to boot, load weights, and warm up CUDA kernels. Traditional CPU-based autoscalers react too slowly. Implement predictive scaling based on request queue depth trends and historical patterns. Maintain a buffer of 1-2 warm replicas during peak hours. Use KEDA with custom metrics from your vLLM Prometheus exporter to trigger scale-up when p95 queue wait exceeds 2 seconds, and scale-down only after 10 minutes of sustained low utilization to avoid thrashing.

Optimizing Serve LLMs in Production: Throughput and Latency as a Continuous Practice

Serving LLMs reliably is not a one-time configuration exercise. Model updates change memory profiles, traffic patterns shift seasonally, and new runtime optimizations emerge monthly. Establish a quarterly review cycle: re-benchmark against latest runtimes, validate quantization quality after base model upgrades, and right-size GPU allocations based on observed utilization. Automate evidence collection for these reviews if you operate under SOC 2 or ISO 27001 compliance frameworks.

Your next step should be concrete: deploy vLLM with the configuration above on a single GPU node, run the three-scenario load test suite, and establish baseline TTFT/TPS numbers before scaling horizontally. If you need help designing a production-grade LLM serving architecture tailored to your workload and compliance requirements, reach out to discuss your specific deployment.

Frequently Asked Questions