The LLM Gateway & Cost Optimization Playbook

The moment your application talks to more than one model — or even one model in more than one place — you have an infrastructure problem hiding in plain sight. Which provider? What happens when it returns a 529? How do you stop a runaway loop from burning $4,000 overnight? How do you prove no PII left the building? The answer to all of these is a gateway: a single layer every LLM call flows through. This playbook is the complete knowledge base for designing one.

Choosing a gateway? See the companion comparison: OpenRouter vs LiteLLM vs Portkey. This playbook is about the architecture and policy a gateway should enforce, whichever you run.

1. What a Gateway Is and Why You Need One

A gateway is a proxy that sits between your application and every model provider. Instead of your code calling OpenAI, Anthropic, and a local model directly, it calls the gateway, and the gateway decides what actually happens.

text
                      ┌──────────────── LLM GATEWAY ────────────────┐
  Your app ──▶ one API│ auth → routing → caching → guardrails →     │──▶ Provider A
                      │ retries → fallback → logging → cost metering │──▶ Provider B
                      └──────────────────────────────────────────────┘──▶ Local model

What it buys you:

• One interface across many providers (usually OpenAI-compatible), so swapping models is a config change, not a code change.
• Reliability — automatic retries and fallback to another provider when one degrades.
• Cost control — budgets, caching, and routing cheap-first.
• Safety — guardrails, PII redaction, and audit trails in one enforceable place.
• Observability — every call logged with latency, tokens, and cost.

Without a gateway these concerns get copy-pasted (badly) across every service that calls a model.

2. Routing Strategies

Routing is the gateway's core job: given a request, which model handles it? The strategies, from simple to sophisticated:

• Static. Always route to model X. Fine to start; brittle when X has an outage.
• Fallback chains. Try model A; on error or timeout, fall to B, then C. The baseline for reliability.
• Load balancing. Spread traffic across providers/keys (round-robin, least-latency) to dodge rate limits and reduce tail latency.
• Cost-tiered routing. Send the request to a cheap model first; escalate to a premium model only when the cheap one is low-confidence or fails a check.
• Semantic / capability routing. Inspect the request and route by need — a code task to a coding-strong model, a quick classification to a tiny fast one.
• Conditional routing. Route on request metadata (customer tier, region, data sensitivity).

yaml
# Illustrative fallback + cost-tiered policy
route:
  primary:   { model: small-fast,    max_cost_per_1k: 0.20 }
  escalate_if: { confidence_below: 0.7, or_error: true }
  fallback:
    - large-quality
    - secondary-provider/large-quality
  timeout_ms: 8000

3. Reliability Engineering

Providers fail. Plan for it at the gateway so your app never has to.

• Timeouts. Every call needs an upper bound. A hung provider should fail fast into a fallback, not block a user for 30 seconds.
• Retries with backoff. Retry transient errors (429, 5xx) with exponential backoff and jitter — but cap attempts so you don't amplify an outage.
• Circuit breakers. When a provider's error rate or latency crosses a threshold, remove it from rotation automatically, then probe it periodically and reinstate when healthy.
• Idempotency. Use request keys so a retry doesn't double-charge or double-act.
• Graceful degradation. Define what happens when all providers fail — a cached answer, a queued retry, or an honest error beats a spinner.

text
Provider error rate > 25% over 1 min  ──▶ OPEN circuit (stop routing here)
        │
   periodic health probe succeeds      ──▶ HALF-OPEN (trial traffic)
        │
   trials succeed                       ──▶ CLOSE circuit (resume)

4. Cost Optimization

This is where a gateway pays for itself. Inference cost is the line item that surprises finance.

A. Semantic caching

Cache responses keyed by meaning, not exact string. If two users ask the same question in different words, serve the cached answer. For FAQ-style and retrieval workloads this can cut spend dramatically.

B. Model tiering

The biggest model is rarely needed for the average request. Default to a small/cheap model and escalate only the hard cases. A well-tuned tier split often moves the majority of traffic to a model costing a fraction of the premium one.

C. Prompt and token discipline

• Trim bloated system prompts — you pay for every token on every call.
• Use prompt caching (where the provider supports it) so static instructions aren't reprocessed.
• Cap max_tokens to what you actually need; unbounded output is unbounded cost.

D. Batch the non-urgent

Anything that can tolerate latency — summarization, enrichment, evals — should run on discounted batch tiers rather than real-time endpoints.

E. Budgets and kill-switches

Set hard per-key, per-customer, and per-day budgets at the gateway. A bug or abuse should hit a ceiling, not your bank account.

5. Observability

You cannot optimize or debug what you can't see. The gateway is the natural place to capture, for every request:

• Model, provider, region, and which route fired.
• Input/output token counts and computed cost.
• Latency (queue, time-to-first-token, total).
• Cache hit/miss, retries, and fallbacks triggered.
• A trace ID linking the call to the user-facing action.

Aggregate this into dashboards for cost per feature, latency P95 per route, and fallback rate per provider — the three views that drive most optimization decisions.

6. Guardrails and Safety

Centralizing safety at the gateway means it's enforced consistently, not reimplemented per service.

• PII redaction on the way in (and out), so sensitive data never reaches a provider or a log it shouldn't.
• Prompt-injection / jailbreak detection before the request hits the model.
• Output filtering for policy violations before a response reaches the user.
• Audit trails — an immutable record of what was sent and returned, essential for compliance reviews.

Wire guardrails into CI: a prompt template that leaks PII or trips a jailbreak check should fail the build before it ships.

7. Self-Hosted vs Managed

The architecture is the same; the operational model differs.

• Managed (SaaS marketplace): zero infrastructure, instant multi-model access, but added latency and less routing control. Best for prototyping and small-to-medium loads.
• Self-hosted (open-source proxy): full control over routing and data, minimal added latency, no lock-in — but you own uptime, scaling, and patching.
• Hybrid: run a self-hosted proxy for production traffic while using a managed marketplace for experimentation and overflow.

See the gateway comparison for how specific tools land on this spectrum. The decision usually reduces to two questions: do you need to self-host? and do you need built-in guardrails?

8. A Phased Rollout Plan

1. Wrap, don't rewrite. Point existing calls at the gateway with a single static route. Change nothing else. Confirm parity.
2. Add observability. Turn on per-request logging of cost, tokens, and latency. Find out where the money goes.
3. Add reliability. Introduce timeouts, retries, and a fallback chain. Verify with a chaos test (kill the primary provider).
4. Optimize cost. Layer in model tiering, caching, and budgets, watching quality metrics so you don't trade accuracy for pennies.
5. Enforce safety. Add PII redaction, guardrails, and audit logging; gate deploys on them.

9. The Production Checklist

• All LLM traffic flows through the gateway — no direct provider calls in app code.
• Every call has a timeout; transient errors retry with capped backoff.
• A fallback chain is defined and tested by killing the primary provider.
• Circuit breakers remove unhealthy providers and auto-reinstate them.
• Cheap-first model tiering routes the majority of traffic to a low-cost model.
• Semantic caching is enabled for repetitive workloads.
• Per-key/customer/day budgets and max_tokens caps are enforced.
• Every request is logged with model, tokens, cost, latency, and a trace ID.
• PII redaction, guardrails, and audit trails are active and CI-gated.
• Dashboards exist for cost-per-feature, P95 latency, and fallback rate.

This playbook pairs with our LLM gateway comparison. Architect the policy here; choose the tool there.