LLM Model Routing and Cascading: Cut Cost Without Cutting Quality (2026)

By Khimananda Oli | Last reviewed: October 2026

Why send every query to the same model in the first place?

LLM model routing and cascading both start from the same observation. If you have already built an AI gateway, the question of which model to call for a given request is one the gateway traffics but does not answer for you — see building an AI gateway for LLM routing for the infrastructure layer: fallback chains, observability, deployment. This article is about the decision underneath that infrastructure: given a request, which model should actually answer it, and when should you escalate.

The case for not sending everything to your best model is almost entirely about cost, and the scale of the gap is larger than intuition suggests. A frontier model and a mid-tier model both correctly answer "what's the capital of France" — the frontier model just costs more to do it. Across a real production traffic mix, a large share of queries are exactly this simple: lookups, short classifications, straightforward rewrites, FAQ-shaped questions. Routing those to a cheaper model and reserving the frontier model for what actually needs it is not a quality compromise if the routing decision is good — it is removing spend that was never buying anything.

What do routing and cascading actually do differently?

Both patterns combine a cheap, fast model with a stronger, slower, more expensive one. They differ in when the decision happens and what information it has access to.

The two are not mutually exclusive. Published research on "cascade routing" formalises combining them in one framework — route first on a cheap signal, then cascade only the queries the router itself is uncertain about — and reports this beats either approach alone. In practice, most teams start with one of the two in isolation, because each is simpler to reason about and debug independently.

What does RouteLLM's actual research show?

RouteLLM is one of the better-documented published approaches to routing, and its reported numbers are specific enough to reason about rather than take on faith. The paper trains routers on 80,000 pairwise human preference battles from Chatbot Arena — pruned to roughly 65,000 after removing very short prompts — and augments that with MMLU-derived gold labels and a GPT-4-judged dataset of around 120,000 additional samples. It evaluates four router architectures: similarity-weighted ranking, matrix factorization, a BERT classifier, and a causal LLM classifier (Llama 3 8B), routing between a strong model (GPT-4) and a weak model (Mixtral 8x7B).

Read the MT-Bench row carefully: a trained router sent only 13.4% of queries to GPT-4 and still retained 95% of GPT-4's measured quality on that benchmark — a roughly 3.7x cost-saving ratio compared to randomly routing half the traffic to each model. The pattern across all three benchmarks is the same shape even though the exact numbers differ: a well-trained router identifies a large share of queries the weak model handles just as well, and spends the expensive model's budget on the minority that actually needs it. The paper also reports the trained routers transfer reasonably to other model pairs — Claude 3 Opus/Sonnet, Llama 3.1 70B/8B — without retraining, which matters if you expect to swap providers later.

The honest caveat is that these numbers are benchmark-specific and depend on having representative preference data for your own traffic. A router trained on Chatbot Arena battles is tuned to the kind of queries people send chatbots in the open, not necessarily your support tickets or your code-review comments.

How do you build a routing layer without training a custom classifier first?

You do not need Chatbot Arena-scale preference data to get real value — a heuristic router, built from signals you already have, captures a meaningful share of the saving with a day of work instead of a training pipeline.

import re

def route(query: str, context_tokens: int = 0) -> str:
    """Cheap heuristic router. Escalate on any signal that a
    small model is known to struggle with; default to cheap."""

    # Long context or multi-document reasoning: send to the strong model
    if context_tokens > 4000:
        return "strong"

    # Code generation and multi-step math are classic small-model failure modes
    code_signals = r"```|\bdef \w+\(|\bclass \w+|SELECT .+ FROM"
    math_signals = r"\bsolve\b|\bprove\b|step[- ]by[- ]step"
    hard_pattern = re.search(code_signals, query, re.I) \
        or re.search(math_signals, query, re.I)
    if hard_pattern:
        return "strong"

    # Short, factual-looking queries are the cheap model's sweet spot
    if len(query.split()) < 40:
        return "cheap"

    return "cheap"  # default: most traffic is simpler than it looks

Two things to get right before this is production-safe. First, log every routing decision alongside which model actually handled the query and, where you can get it, a quality signal — a thumbs up/down, a follow-up clarifying question, a support escalation. That log is what eventually lets you replace the heuristic with a trained router, and it is also what tells you the heuristic is wrong before a customer does. Second, treat the thresholds as configuration, not constants — a 40-word cutoff that works for a support bot will misroute a coding assistant's traffic almost entirely.

How do you build a cascade, and what stops it from escalating everything?

A cascade needs a cheap way to decide whether the cheap model's answer is good enough to keep — and that decision is the part teams get wrong most often, because it is tempting to make it too lenient or too strict without measuring either.

def answer_with_cascade(query: str, cheap_model, strong_model,
                         confidence_threshold: float = 0.7) -> str:
    cheap_response, confidence = cheap_model.generate_with_confidence(query)

    if confidence >= confidence_threshold:
        return cheap_response

    # Escalate: the cheap model itself is not confident, or a verifier
    # (self-consistency check, a classifier, a cheap judge model) flagged it
    return strong_model.generate(query)

Confidence can come from several places, in increasing order of cost and reliability: the model's own token-level log-probabilities (cheapest, noisiest), a self-consistency check — generate the cheap answer twice with different sampling and escalate on disagreement — or a small separate verifier model trained or prompted specifically to judge "does this answer fully address the query." A 2026 line of research on quality-estimator-free cascade routing exists specifically because training a good confidence estimator is its own hard problem; if you do not have one yet, self-consistency is the cheapest honest starting point.

The documented failure mode worth planning around from day one: a query gets classified as simple, the cheap model produces something confidently wrong and unhelpfully brief rather than visibly failing, and a weak confidence signal lets it through. This is why a cascade needs more than a single scalar threshold — pair it with length-aware and format-aware checks (a one-sentence answer to a question that clearly needs a structured breakdown is a decent escalation signal on its own) rather than trusting confidence alone.

What does the cost math actually look like at scale?

A concrete comparison makes the trade-off between routing and cascading less abstract. Take a support assistant handling 1 million queries a month, where a cheap model costs roughly 1/15th the price of a frontier model per request, and assume — consistent with RouteLLM's MT-Bench result — that 85% of real traffic is genuinely simple enough for the cheap model to match the frontier model's quality.

At this traffic split the two strategies land close together on pure model spend — cascading's extra cost is the discarded cheap-model attempt on every escalated query, which is small per call but adds up across 150,000 escalations. The real difference shows up in latency and architecture, not the invoice: routing adds a lookup before the single model call it needs, while cascading adds the full round-trip time of the cheap model's attempt to every escalated query's total response time. For a chat interface where users notice a slow reply, that added latency on the hardest 15% of queries is often the deciding factor, not the marginal dollar difference.

How do you know an LLM model routing setup is actually working?

Routing and cascading are both invisible failures until someone measures them — a misrouted query does not throw an error, it just produces a worse answer that looks like a normal response. Treat this the same way you would any other agent or LLM change: with the evaluation discipline covered in evaluating AI agents, applied to a narrower question — did routing this specific query to the cheap model produce an answer of comparable quality to what the strong model would have given? Running a sample of production traffic through both paths periodically, and diffing the outcomes, is the only reliable way to catch a router whose accuracy has drifted as your traffic mix has shifted.

Instrument both patterns the same way you would any LLM call — tag the model actually used, the routing or cascade decision, and the outcome, using OpenTelemetry's GenAI semantic conventions. gen_ai.request.model and gen_ai.response.model already distinguish the two cleanly in a cascade, since the final response model differs from the first attempt; add a custom attribute for which router decision fired so you can later correlate routing choices with the task-completion and cost dashboards from LLM cost optimization for production apps. Without that correlation, a routing layer that is quietly costing you more in rework than it saves in API spend looks identical, from the billing dashboard alone, to one that is working exactly as intended.

If you are weighing providers for either the cheap or strong leg of a routing setup, choosing an LLM API: cost, speed, quality covers the selection criteria this article assumes you have already settled.

If you want a routing or cascading layer designed against your own traffic — with the right heuristics, the measurement to prove it is not degrading quality, and a path to a trained router once you have the data — my DevOps and cloud consulting services cover exactly this.