Multi-Provider AI Fallback: Reliability Beyond Retries

An AI feature is not available just because its provider returns HTTP 200. It is available when the user gets a usable result before giving up.

That distinction shaped the routing layer in Codelit. A second provider helps with outages. It does not fix invalid output, an exhausted budget, or a fallback model that cannot do the job.

Fallback reduces dependence on a single provider's availability, capacity, or model behavior. It does not remove shared dependencies or guarantee equivalent answers. The router itself, your network, and the validation layer remain part of every request's failure path.

The textbook calculation is useful only with its assumptions attached. Two providers that are each available 99% of the time would offer 99.99% combined availability if failures were independent and switching always worked. Real providers can share cloud regions, upstream models, and rate-limit bottlenecks.

I would publish an uptime figure only with its measurement window, latency threshold, and definition of success. Counting successful HTTP responses makes the dashboard look better while hiding broken diagrams.

A provider registry should describe capabilities, policy constraints, pricing inputs, and operational state. Model names belong in configuration, not scattered through UI code. Each entry should be specific enough that an incompatible provider cannot enter a fallback chain accidentally.

Codelit's routing work reinforced the same idea behind ResolveMesh: vendor independence is an interface boundary, not a claim that all vendors are interchangeable.

Model retirement belongs in the maintenance checklist. Replace registry entries deliberately, then rerun the task evaluations before sending production traffic to the replacement. A new model name is a configuration change; compatibility is something to verify.

A complexity scorer can make an inexpensive first routing decision, but it is only an estimate. A short request can be difficult, and a long prompt can contain simple background material. Evaluate the scorer against the work the product actually performs.

For architecture generation, useful signals include explicit consistency requirements, latency constraints, regions, and dependency count. I would use those signals to nominate a starting tier, then compare that choice with an evaluation set.

The success criterion is not "the cheaper model returned JSON." It is whether the result passes the same task-level checks as the alternative. That validation is part of the prompt-to-diagram pipeline, not a substitute for it.

Retry transient failures when another attempt is likely to help and the operation is safe to repeat. Treat invalid input, missing permissions, and policy refusals separately. Switching providers should not be a way to evade a restriction or silently weaken the output contract.

A retry chain needs an overall deadline. Four ten-second timeouts can turn one failed request into a forty-second wait. Stop when the user cancels, the deadline expires, or the remaining allowance cannot cover another attempt.

For streaming responses, inspect the event stream as well as the status code. Anthropic documents errors arriving inside an already-open stream. That is why progressive rendering needs an explicit incomplete state.

Provider adapters should normalize the fields the application needs while preserving important differences such as tool calls, refusals, finish reasons, and missing usage. Flattening every response into a string makes integration easy until the discarded information becomes necessary for correctness.

Use an internal result that includes content, provider, model, finish state, usage when available, and request identifiers. Keep raw diagnostic payloads only under a redaction and retention policy.

Do not assume one universal "OpenAI shape." Chat Completions and Responses use different contracts. Likewise, a Messages response can contain several blocks, not just text in its first element.

OpenRouter documents an OpenAI-compatible response format, not an arbitrary extra wrapper around each upstream provider. The adapter should follow the API you called, not the vendor you imagine sits behind it.

A health check measures a narrow path through a dependency. Real request outcomes provide the broader signal. Combine both in a circuit breaker, and use limited recovery traffic to find out whether the provider can handle the workload that previously failed.

A tiny text prompt does not test long contexts, structured outputs, tool calls, or a particular account's quota. Track failures by model and capability where the distinction matters.

Recovery thresholds are tuning parameters, not universal constants. Choose them from observed failure patterns and the cost of a false recovery. A probe that succeeds once is a reason to test carefully, not proof that the outage is over.

Cost and rate limits belong in request admission, not only in reporting. Every fallback attempt consumes shared capacity and may incur a charge even when its output is discarded. Per-user limits alone cannot protect a provider account shared by many users.

Track cost per successful task, including failed attempts, repairs, and tool charges. Break it down by task class before using it to make pricing decisions.

Use separate controls for tenant rate, account rate, concurrency, and spend. A counter keyed by user and date is a fixed daily quota, not a sliding-window limiter. Both can be useful, but they behave differently at a window boundary.

I would start with one primary provider, one evaluated fallback, a shared deadline, and a trace of every attempt. That is enough to test whether fallback improves completed work before expanding the registry or building a sophisticated routing score.

Then test the unattractive cases: slow responses, invalid JSON, missing usage, mid-stream errors, cancellation, and both providers unavailable. Keep a clear failure message for the final case.

The point is not to say the product uses more models. The point is to keep one dependency's bad day from becoming the user's dead end, without hiding what changed along the way.