Cruxly › AI engineering › LLM Backend Infrastructure

Multi-Provider Failover That Actually Fails Over

circuit breakers, fallback chains, and the flapping trap

Prompt: design the request path for an API that calls a foundation model, needs to survive a provider outage or sustained rate-limiting, and must not flap uncontrollably between providers on ordinary transient errors.

40 minutes first. The naive version — "if provider A fails, call provider B" — is a real trap disguised as a solution.

Why a bare try/except is not a fallback chain

The naive version fails over on the wrong signal. Catching every exception and calling a second provider means a single slow-but-successful request (one that technically completes but took 9 seconds) either isn't caught at all, or — if you also catch timeouts — one occasional slow request switches the entire system to a different model for reasons no one decided. Both outcomes are unacceptable at production volume: the first hides a real problem, the second switches the whole system on noise.

Health must be tracked per provider, over a window, not per individual request. A circuit breaker maintains a rolling count of recent successes and failures for each provider. It opens (stops sending traffic to that provider) only after a real pattern emerges — say, 5 failures in the last 20 requests — not after a single blip that could be pure noise.

Three states, not two. Closed (normal, sending traffic), Open (provider looks unhealthy, traffic redirected to the fallback), and Half-Open (cautiously sending a small fraction of traffic back to test if the provider recovered, without committing fully until it's confirmed). Skipping Half-Open means either the system never recovers automatically (Closed/Open only, requiring a manual reset) or it recovers too eagerly and flaps.

The fallback chain: same contract, different provider

A fallback provider must honor the exact same response contract as the primary, or the fallback itself becomes a new failure mode. If your code expects a specific JSON shape from provider A's structured output feature and provider B formats structured output differently, the fallback "works" (the call succeeds) while silently breaking every downstream consumer of that malformed response. Normalize provider responses into one internal shape at the boundary, before any business logic sees them.

Order the fallback chain by more than just preference. Cost, latency profile, and capability (does this provider support the same context length, the same structured output mode, the same multimodal input) all matter. A fallback to a cheaper-but-slower provider under load is a different engineering decision than a fallback to a faster-but-pricier one, and the chain's order should reflect which trade-off you're making, stated explicitly rather than left as an artifact of which provider you integrated first.

Prefer the same model on another cloud before a different model family. Many models are served by more than one platform (the vendor's own API plus one or more cloud marketplaces). Failing over to the same model elsewhere keeps behaviour, prompts and evals unchanged. Failing over to a different model family changes all three.

Prompts aren't portable, so the fallback path needs its own evals. A prompt tuned on one model can behave differently on another — different instruction-following, different formatting, different refusal behaviour. Run your eval suite against every model in the chain, not only the primary, or the first real failover is also the first time anyone sees how the fallback actually answers.

A failure mid-stream can't fail over transparently. If provider A has already streamed 300 tokens to the user when it errors, provider B can't seamlessly continue that exact text. Decide the policy explicitly: show an error and offer a retry, or restart the answer on the fallback and tell the user the response was regenerated. Fallback chains work cleanly only for failures before the first token.

Log every failover event with enough detail to reconstruct why it happened. Which provider failed, what the error was, how long the circuit stayed open, when it recovered. Without this, "why did costs spike Tuesday" or "why did users get worse answers for twenty minutes" are unanswerable after the fact.

Avoiding the flapping trap

Failing back to the primary provider needs its own threshold, distinct from the threshold that opened the circuit. If a provider recovers and you switch back immediately on the first success, a provider that's intermittently healthy (succeeds once, fails again) causes rapid oscillation between providers — each switch potentially changing model behavior and cost mid-session for a user. Require sustained health (several consecutive successes, or a cooldown period) before fully closing the circuit again.

Jittered exponential backoff on retries, always — never a fixed retry delay. A fixed delay means every client retrying after a shared outage retries at the exact same moment, recreating the load spike that caused the outage in the first place. Jitter (randomizing the delay within a range) spreads retries out over time instead of synchronizing them.

Not every error should be retried, and the provider tells you how long to wait.

• Honour Retry-After on 429 and 503 responses instead of your own backoff schedule — the provider knows when capacity returns.
• Don't retry most 4xx errors. A 400 (bad request), 401/403 (auth) or 413 (too large) will fail identically on every retry; retrying them is how a validation bug becomes a cost spike.
• Treat provider "overloaded" responses as their own signal (some providers use a dedicated status code for this). They're transient capacity problems, a good trigger for the breaker, and a reason to try the fallback rather than hammering the primary.
• Use two timeouts, not one. A time-to-first-token timeout catches a stuck request in seconds; a separate, much longer total timeout allows a reasoning model that legitimately thinks for minutes to finish. A single 30-second timeout either kills reasoning models or waits far too long to notice a hung connection.

A circuit breaker per provider and per capability, not one global breaker. A provider might be healthy for short completions and struggling specifically with long-context requests under load — a single breaker that trips on aggregate error rate misses this and either over-reacts (routing away from a provider that's fine for most traffic) or under-reacts (missing a real degradation in one specific capability).

What to monitor, and the alert that actually matters

Track per-provider: success rate, p50/p95/p99 latency, and circuit state, continuously — not just "is it up." A provider that's technically up but degraded to 8-second p95 latency is failing your users even though every health check returns 200.

Alert on sustained circuit-open state, not on individual failures. Individual failures are expected and handled by the breaker itself; a human needs to know when the breaker has been open for an extended period, because that means the fallback provider is now carrying 100% of traffic and its capacity and cost profile need attention too.

The most commonly missed alert: fallback provider approaching its own limits. If the primary is down and all traffic shifted to the fallback, the fallback can now hit rate limits or budget caps it was never sized for under normal conditions — a cascading failure where fixing the first outage causes a second one, unless the fallback's own headroom is monitored during a failover, not just under normal load.

How to say it in the interview

"A single try/except around the API call that falls back to a second provider on any exception looks like resilience, but it means one slow response, not just an outright failure, silently switches every subsequent request to a different model with different behavior, different cost, and no one decided that on purpose. The fallback needs its own decision rule, not just an error handler."

Checklist

• Tracked provider health as a rolling window, not per-request pass/fail
• Implemented Closed/Open/Half-Open, not just a binary healthy/unhealthy flag
• Normalized every provider's response into one internal shape at the boundary
• Ordered the fallback chain by cost, latency, and capability, stated explicitly
• Required sustained health before closing the circuit back to the primary
• Used jittered exponential backoff, never a fixed retry delay
• Ran a circuit breaker per provider per capability, not one global breaker
• Honoured Retry-After, skipped retries on non-retryable 4xx, and used separate TTFT and total timeouts
• Preferred same-model multi-cloud fallback, ran evals on every fallback model, and set a mid-stream failure policy
• Monitored the fallback provider's headroom specifically during a failover

Related lessons

• Cost Attribution and Runaway Prevention
• Prompt Caching: Paying for the Same Tokens Once