Model cascades and tiered LLM routing in production APIs

Your support assistant works. Users get helpful answers. Finance forwards the invoice anyway: every question—whether a one-line password reset or a multi-step billing dispute—runs through the same frontier model at the same price tier. Latency p95 climbs on simple queries because they wait behind long, expensive completions. Meanwhile, a smaller model on your staging stack answers eighty percent of the traffic correctly in benchmarks you never wired into production routing.

This article is about model cascades: deliberately ordering models from cheap to capable, escalating only when cheaper tiers fail a quality gate. The goal is not to starve hard questions of intelligence; it is to stop paying frontier prices for work a smaller model already handles reliably. These patterns show up in almost every consulting engagement where LLM features graduate from demo to line item: predictable cost, latency, and quality floors matter as much as raw capability.

What a model cascade is (and what it is not)

A model cascade sends a request through one or more tiers in sequence or in parallel until a tier produces an answer that passes your acceptance criteria—or you exhaust the ladder and fail gracefully.

• Not random load balancing across models with different prices. That optimizes utilization, not unit economics.
• Not a replacement for semantic caching or RAG. Caching reuses prior outputs; retrieval grounds answers in documents; cascades choose which model should attempt the work.
• Not free complexity. You pay engineering time for routing logic, evaluation, and the operational surface area of multiple providers and model versions.

A typical three-tier ladder:

Tier	Role	Example models (illustrative)
0 — Router	Cheap classification or rules	Small classifier, keyword heuristics, embedding similarity to known intents
1 — Workhorse	Default completion path	Mid-size instruct model
2 — Escalation	Hard reasoning, long context, tool-heavy flows	Frontier model

The cascade starts at tier 1 (or tier 0 if you have a strong router) and escalates when the output fails validation, confidence is below threshold, or the router flags complexity.

When cascades earn their keep

Cascades pay off when your traffic has heterogeneous difficulty and your pricing is heterogeneous by model:

• Support and ops copilots — large volume of FAQs and templated workflows; long tail of exceptions.
• Classification and extraction — many rows are easy (intent: password_reset); some need nuanced parsing.
• Code assistants with guardrails — simple refactors vs. architectural questions.
• Multi-tenant SaaS — free tier capped to smaller models; paid tier escalates automatically.

They are weaker when:

• Every request truly needs frontier quality — legal review, high-stakes medical triage (with human oversight), or tasks where a single mistake is unacceptable. Cascades still help with latency via speculative parallel paths, but cost savings shrink.
• Your evaluation cannot distinguish tiers — if you cannot measure “good enough,” routing becomes guesswork.
• Latency SLO is tighter than the sum of tiers — sequential escalation adds wall-clock time unless you design carefully (see below).

Routing signals: how to decide tier and escalation

Static rules and heuristics

Fast, explainable, brittle:

• Token count or attachment size over threshold → skip tier 1.
• Presence of structured tool calls required → start at tier with reliable function-calling.
• Tenant plan / feature flag → cap maximum tier.

Use rules for hard constraints, not for nuanced quality judgment.

Classifier models

A small model (or embedding + logistic regression) predicts complexity, intent, or escalation probability from the user message and light metadata (locale, product area, prior turns). Train or prompt-evaluate on production-like examples; label not only “correct answer” but which tier sufficed.

Trade-off: classifier drift when product vocabulary changes—retrain or refresh few-shot examples on a schedule.

Self-consistency and verifier passes

After tier 1 answers, a cheap verifier checks:

• JSON schema validity (structured outputs).
• Citation coverage — every factual claim must map to a retrieved chunk id.
• NLI-style entailment — does the answer contradict retrieved context?

Failed verification triggers escalation. This pattern separates generation cost from verification cost and is often cheaper than always calling the frontier model.

Confidence from the model itself

Some APIs expose logprobs or calibrated scores; many do not reliably. Treat model-reported confidence as a weak signal unless you have calibrated it on your data. Prefer external checks (schema, retrieval overlap, verifier model) for production gates.

Architecture patterns

Sequential escalation (default)

Request → Tier 1 → [pass?] → return
                 → [fail]  → Tier 2 → [pass?] → return
                                    → [fail]  → error / human / Tier 3

Pros: minimal spend when tier 1 succeeds. Cons: adds latency on misses (two serial model calls).

Mitigations:

• Short-circuit timeouts — if tier 1 exceeds a latency budget, abort and escalate rather than waiting for a full completion.
• Speculative parallel start — begin tier 2 only when tier 1 verifier runs; or run tier 1 and retrieval in parallel while tier 0 routes (see below).

Parallel race with arbitration

Start tier 1 and tier 2 together; return the first that passes verification, cancel the other. Useful when miss rate is high but latency SLO is strict.

Cost warning: you pay for both on many requests—justify with math on (miss_rate × dual_cost) vs (sequential_latency × churn).

Router-first (tier 0)

Tier 0 outputs { target_tier, reason } in a structured schema. Downstream calls only the chosen model. Good when escalation is expensive and you can predict difficulty upfront.

Combine tier 0 with sticky routing per session: after one escalation, subsequent turns in the same thread may start at tier 2 if context is already large.

Quality gates: define “good enough” per route

Each API route should declare an explicit acceptance contract:

Route	Gate examples
POST /support/reply	Answer length bounds; must include citation_ids[] when RAG used; toxicity score below threshold
POST /extract/invoice	JSON schema; numeric fields reconcile to line items
POST /code/suggest	Syntax-valid for declared language; no secret-like tokens

Escalation triggers should be enumerable in logs (escalation_reason: verifier_citation_missing) so you can tune thresholds without mystery spend.

Observability and cost attribution

Instrument every hop:

• llm.tier (0, 1, 2), llm.model, llm.provider
• llm.input_tokens, llm.output_tokens, llm.estimated_cost_usd
• llm.escalated (boolean), llm.escalation_reason
• trace_id propagated from HTTP handler through verifiers

Dashboards worth building:

• Cost per successful request by route and tenant—not just total spend.
• Escalation rate over time; spikes often precede product or prompt changes.
• Tier-1 pass rate on golden eval sets after each prompt or model version change.

In production systems I help teams harden, cascades fail operationally when nobody can answer: “Why did we spend 3× yesterday?”—structured escalation reasons fix that.

Interaction with caching, RAG, and sessions

Order of operations that usually works:

1. Auth, rate limits, abuse checks.
2. Semantic cache lookup (optional)—if hit, skip all tiers.
3. Retrieval (if RAG)—shared across tiers; do not re-embed per tier unless prompts differ materially.
4. Cascade generation with shared retrieved context.
5. Write-through cache on success (respect tier in cache key metadata if answers differ by model).

Include model_tier and prompt_version in cache keys when answers must not cross tiers. A tier-1 paraphrase must not satisfy a user who paid for tier-2 quality.

Practical example: sequential cascade with schema verification

The following TypeScript sketch implements a two-tier cascade with a JSON-schema check and a citation verifier for a RAG-style support reply. It uses provider-agnostic types; wire complete() to your OpenAI-compatible client.

import { z } from "zod";

const SupportReplySchema = z.object({
  answer: z.string().min(1).max(4000),
  citation_ids: z.array(z.string()).min(1),
  confidence: z.enum(["low", "medium", "high"]),
});

type SupportReply = z.infer<typeof SupportReplySchema>;

type CompleteFn = (input: {
  model: string;
  messages: Array<{ role: "system" | "user" | "assistant"; content: string }>;
  signal?: AbortSignal;
}) => Promise<{ text: string; usage: { input: number; output: number } }>;

type Chunk = { id: string; text: string };

const MODELS = {
  workhorse: "gpt-4o-mini",
  escalation: "gpt-4o",
} as const;

function buildMessages(userQuestion: string, chunks: Chunk[]) {
  const context = chunks.map((c) => `[${c.id}] ${c.text}`).join("\n\n");
  return [
    {
      role: "system" as const,
      content:
        "Answer using only the provided context. Respond with JSON: " +
        '{ "answer": string, "citation_ids": string[], "confidence": "low"|"medium"|"high" }',
    },
    {
      role: "user" as const,
      content: `Context:\n${context}\n\nQuestion: ${userQuestion}`,
    },
  ];
}

function parseReply(text: string): SupportReply | null {
  try {
    return SupportReplySchema.parse(JSON.parse(text));
  } catch {
    return null;
  }
}

function verifyCitations(reply: SupportReply, chunks: Chunk[]): string | null {
  const valid = new Set(chunks.map((c) => c.id));
  const missing = reply.citation_ids.filter((id) => !valid.has(id));
  if (missing.length > 0) return `unknown_citation_ids:${missing.join(",")}`;
  if (reply.confidence === "low") return "low_confidence";
  return null;
}

export async function supportReplyCascade(
  complete: CompleteFn,
  userQuestion: string,
  chunks: Chunk[],
  signal?: AbortSignal,
): Promise<{
  reply: SupportReply;
  model: string;
  escalated: boolean;
  escalation_reason?: string;
  usage: { input: number; output: number };
}> {
  const messages = buildMessages(userQuestion, chunks);
  let totalUsage = { input: 0, output: 0 };

  const runTier = async (model: string) => {
    const result = await complete({ model, messages, signal });
    totalUsage = {
      input: totalUsage.input + result.usage.input,
      output: totalUsage.output + result.usage.output,
    };
    const parsed = parseReply(result.text);
    if (!parsed) return { ok: false as const, reason: "invalid_json" };
    const citeErr = verifyCitations(parsed, chunks);
    if (citeErr) return { ok: false as const, reason: citeErr, partial: parsed };
    return { ok: true as const, reply: parsed };
  };

  const tier1 = await runTier(MODELS.workhorse);
  if (tier1.ok) {
    return { reply: tier1.reply, model: MODELS.workhorse, escalated: false, usage: totalUsage };
  }

  const tier2 = await runTier(MODELS.escalation);
  if (tier2.ok) {
    return {
      reply: tier2.reply,
      model: MODELS.escalation,
      escalated: true,
      escalation_reason: tier1.reason,
      usage: totalUsage,
    };
  }

  throw new Error(`cascade_exhausted:last_reason=${tier2.reason}`);
}

Extend this with:

• Per-tier timeouts via AbortSignal and AbortSignal.timeout().
• Token budgets — reject tier 1 early if context exceeds workhorse window.
• Metrics — increment llm_escalation_total{reason=...} when tier 2 runs.

The handler should map cascade_exhausted to a safe user-facing fallback (human handoff, “try again,” or narrowed retry)—not a raw 500 with provider errors.

Common mistakes and pitfalls

1. Escalating without recording why — you cannot tune thresholds or explain invoices.
2. Using the same prompt for every tier — tier 1 may need stricter JSON instructions; tier 2 may need room for reasoning. Blind duplication wastes tokens.
3. Skipping evaluation per tier — offline benchmarks on the frontier model do not prove tier 1 is safe for your gate.
4. Sequential cascade under strict p95 — double model latency on misses; add timeouts, parallel verification, or router-first tier 0.
5. Cache pollution across tiers — storing a tier-1 answer that a paid user would expect from tier 2.
6. Classifier trained on stale intents — product rename causes mass escalation or wrong-tier answers; monitor escalation rate alerts.
7. Treating escalation as failure — some routes should expect tier 2 for certain intents; price and capacity-plan accordingly.
8. No kill switch per tier — if a provider degrades one model id, you need flags to bypass or substitute without redeploying the whole cascade.

Conclusion

Model cascades turn LLM integration from a single expensive dial into a tiered system with explicit quality gates: start with the cheapest path that can satisfy your contract, escalate with measurable reasons, and instrument cost and pass rates per route. Combined with caching, retrieval, and session discipline, routing is often the largest lever on unit economics after prompt size itself.

The takeaway for production APIs: define what “good enough” means per endpoint, encode it in verifiers—not vibes—and make escalation observable. That is how teams keep assistants fast and affordable without betting the product on one model price sheet. For platforms where cost, latency, and correctness all sit on the critical path, getting cascade policy right is core work for scalable, production-ready LLM backends—whether you are shipping internally or helping a team move from pilot spend to sustainable operations.