Multi-Provider AI Fallback: Routing Across 11 Providers

On March 14th, 2025, OpenAI went down for about 4 hours. I know this because my error tracking dashboard went solid red and my product became a very expensive loading spinner.

Codelit, the architecture tool I build solo, was 100% dependent on OpenAI at the time. No fallback. No retry to a different provider. Just a try/catch that showed users "Generation failed, please try again later." They did not try again later. They left.

Multi-provider AI fallback

is a routing strategy that sends every model request down an ordered chain of providers, moving to the next one the moment the current provider is down, rate-limited, or returns output that fails validation.

That outage cost me users I'll never get back. It also gave me the kick I needed to build multi-provider routing. Today Codelit routes across 11 providers with automatic fallback, and our effective uptime for AI features is 99.7% over the last six months.

Here's how the system works.

A single AI provider is a liability because your product's availability can never exceed that provider's, and no model API is reliable enough to bet a business on. One vendor outage becomes a total outage of every AI feature you ship. This isn't theoretical. In the past year:

• OpenAI has had multiple outages lasting 1-4 hours
• Anthropic's API has had capacity issues during peak US business hours
• Google's Gemini API has rate-limited aggressively during high-demand windows
• Smaller providers on OpenRouter go up and down constantly

Each of these providers has great uptime individually, usually 99%+. But 99% uptime means 3.65 days of downtime per year. If your product is the kind of thing people try once and bounce if it doesn't work, even 30 minutes of downtime during a traffic spike can be devastating.

The math changes when you have fallback. If Provider A has 99% uptime and Provider B has 99% uptime, and they fail independently, your effective uptime with fallback is 99.99%. That's 52 minutes per year instead of 3.65 days.

Anything on a schedule has it worse. The twice-daily briefing behind ViralVault is mostly automated, and a publishing slot doesn't wait for a provider to come back.

A multi-provider fallback system needs four parts: a provider registry that knows every model's cost, limits and health; a complexity scorer that decides how much model a request deserves; a routing engine that walks the fallback chain; and a health monitor that keeps the registry honest.

The provider registry is the one table of every provider Codelit can call, holding each one's models, pricing, rate limits, average latency, and current health status.

interface ProviderConfig {
  id: string;
  name: string;
  models: ModelConfig[];
  healthStatus: "healthy" | "degraded" | "down";
  lastHealthCheck: number;
  rateLimits: {
    requestsPerMinute: number;
    tokensPerMinute: number;
    currentUsage: number;
  };
  costPerInputToken: number;
  costPerOutputToken: number;
  avgLatencyMs: number;
}

As of July 2026 the registry spans 11 providers: OpenAI's GPT, Anthropic's Claude, and Google's Gemini direct, plus xAI Grok, Mistral, Groq, Perplexity, Together, DeepSeek, Cerebras, and a free pool through OpenRouter that rotates based on availability. The model list is never fixed. It refreshes as each provider ships new versions.

Keeping the registry vendor-neutral is the same instinct behind ResolveMesh, the capability resolver I launched with a deliberately empty catalog. The interface should never know which vendor is behind it.

The complexity scorer assigns each prompt one of three tiers: simple, moderate, or complex.

Not every request needs a frontier model. If someone asks for a "simple REST API with a database," that's a straightforward architecture. A small free model can handle it fine. But "design a real-time bidding platform with sub-100ms latency, event sourcing, and multi-region failover" is the kind of thing that needs a frontier model.

function scoreComplexity(prompt: string): "simple" | "moderate" | "complex" {
  let score = 0;

  // Length heuristic: longer prompts usually mean more complex requirements
  if (prompt.length > 500) score += 2;
  if (prompt.length > 1000) score += 2;

  // Keyword signals for architectural complexity
  const complexPatterns = [
    /multi.?region/i,
    /event.?sourc/i,
    /cqrs/i,
    /real.?time/i,
    /sub.?\d+ms/i,
    /saga/i,
    /distributed.?transaction/i,
    /consensus/i,
  ];
  score += complexPatterns.filter((p) => p.test(prompt)).length * 2;

  // Component count signals
  const componentKeywords = [
    "microservice",
    "queue",
    "cache",
    "load balancer",
    "CDN",
    "gateway",
    "broker",
    "stream",
  ];
  const componentCount = componentKeywords.filter((k) => prompt.toLowerCase().includes(k)).length;
  score += componentCount;

  if (score >= 10) return "complex";
  if (score >= 4) return "moderate";
  return "simple";
}

This isn't perfect. It's a heuristic. But it's right about 85% of the time, and the fallback chain catches the cases where a simpler model produces bad output. Scoring runs before anything else in the pipeline that turns a prompt into a diagram, because every later stage depends on which model answered.

The routing engine takes the complexity tier and current provider health, then walks the fallback chain in order, skipping anything marked down or rate-limited.

async function routeRequest(prompt: string, userTier: "free" | "pro"): Promise<GenerationResult> {
  const complexity = scoreComplexity(prompt);
  const chain = buildFallbackChain(complexity, userTier);

  for (const provider of chain) {
    if (provider.healthStatus === "down") continue;
    if (provider.rateLimits.currentUsage >= provider.rateLimits.requestsPerMinute) continue;

    try {
      const result = await callProvider(provider, prompt);
      if (validateOutput(result)) {
        return result;
      }
      // Output was structurally invalid, try next provider
      logQualityFailure(provider.id, prompt, result);
    } catch (error) {
      markProviderDegraded(provider.id);
      logProviderError(provider.id, error);
      // Fall through to next provider in chain
    }
  }

  throw new AllProvidersFailedError();
}

The chain order depends on the user's plan and the request complexity. These were the chains when I wrote this in March 2026 — several of the model names have since been retired or renamed, which is the point of the section below:

Every name in that table changed between publishing this in March 2026 and updating it in July. Not one line of the routing did.

Mixtral is the starkest one: it's gone. Mistral retired the 8x7B and 8x22B models and points you at Mistral Small 4 instead, and when I checked OpenRouter's model list on 26 July 2026 there was no mixtral id left on it at all. Llama is on the same path. Groq's deprecation page lists llama-3.3-70b-versatile and llama-3.1-8b-instant shutting down on 16 August 2026, and the replacements it recommends aren't Llama — they're openai/gpt-oss-120b and qwen/qwen3.6-27b.

The paid side moved less violently but it still moved. OpenAI's deprecations page maps gpt-5 to gpt-5.6-sol and gpt-5-mini to gpt-5.6-terra, with the old snapshots leaving the API on 11 December 2026. Anthropic's Haiku and Opus tiers survived under the same names at Haiku 4.5 and Opus 5, and Gemini Flash and Pro are now Gemini 3.6 Flash and Gemini 3.1 Pro.

That churn is the entire argument for the registry. Four months cost me a config file instead of a rewrite, and I only had to touch the column that holds vendor names.

Every chain eventually falls through to cheaper models. A degraded answer beats no answer, and users can always regenerate.

The health monitor pings every provider every 60 seconds with a minimal test prompt and records three things: whether the provider responded, how long it took, and whether the response was parseable.

async function healthCheck(provider: ProviderConfig): Promise<HealthStatus> {
  const start = Date.now();
  try {
    const response = await callProvider(provider, HEALTH_CHECK_PROMPT, {
      timeout: 10000,
      maxTokens: 50,
    });
    const latency = Date.now() - start;

    if (latency > provider.avgLatencyMs * 3) {
      return "degraded"; // Responding but slow
    }
    return "healthy";
  } catch {
    return "down";
  }
}

The probation window is the part people skip, and it's the part that stops a half-recovered provider from taking a fresh wave of requests down with it.

You write one adapter per provider that maps its raw response onto a single internal shape — text, model, provider, token counts, latency, cost — so nothing downstream ever has to know which vendor answered. This was the most annoying part to build. Every provider returns structured data differently.

Each normalizer maps onto the same interface:

interface NormalizedResponse {
  text: string;
  model: string;
  provider: string;
  inputTokens: number;
  outputTokens: number;
  latencyMs: number;
  cost: number;
}

function normalizeOpenAI(raw: OpenAIResponse): NormalizedResponse {
  return {
    text: raw.choices[0].message.content,
    model: raw.model,
    provider: "openai",
    inputTokens: raw.usage.prompt_tokens,
    outputTokens: raw.usage.completion_tokens,
    latencyMs: raw._latency,
    cost: calculateCost("openai", raw.model, raw.usage),
  };
}

Each provider gets its own normalizer. When I add a new provider, I write one normalizer function and add it to the registry. Takes about 20 minutes.

Codelit's AI spend averages $0.003 per free-tier generation and $0.02 per Pro generation, which comes out at roughly $0.30 per active user per month. Every generation logs its cost, tagged by provider, model, complexity tier, and user tier. That split is the interesting part:

Read that table twice. The models handling most of the traffic are not the models spending most of the money, and that inversion is the entire reason the complexity scorer earns its keep.

These numbers inform pricing decisions. The free tier needs to work well enough that people stick around, but the AI cost per free user can't exceed what I'd spend acquiring a paid user. Free tier users cost me about $0.15/month in API calls. That's sustainable if 5% convert to the $5/month Pro plan.

Those two figures are the entire argument for where the paywall sits, and getting them wrong is how I ended up writing about the pricing mistake I made integrating Stripe.

You enforce your own limits on top of theirs, because a provider's rate limit is shared across every one of your users at once. Free users get 10 generations per day. Pro users get unlimited runs. If 50 free users all generate simultaneously, I'll blow through OpenRouter's free tier limits.

I use a sliding window rate limiter backed by Firebase. Each request increments a counter in a document keyed by userId:date. Simple, and it survives server restarts because the state is in the database, not in memory.

Three things stuck from building the router: output validation matters as much as provider selection, latency is part of availability rather than separate from it, and provider diversity paid for itself inside a single outage.

Output validation is as important as provider selection. A provider can respond with a 200 status code and still give you garbage. The AI might return malformed JSON, or valid JSON that doesn't match your schema, or a perfectly structured response that's architecturally nonsensical. My validateOutput function checks structural validity and catches about 3% of responses that would've produced broken diagrams.

Latency budgets matter. Users expect generation to take 3-8 seconds. If a provider is "healthy" but responding in 15 seconds, it feels broken. That's why the health monitor tracks latency, not just availability.

Provider diversity pays for itself. The engineering cost of building multi-provider routing was about two weeks of work, and adding each new provider since has been a normalizer function and a registry entry. The business cost of a 4-hour outage with no fallback was every user who bounced that afternoon and never came back. The math is not close.

If you're building on AI and still using a single provider, build fallback. It doesn't have to be this complex. Even a simple "try OpenAI, if it fails try Anthropic" with a 10-second timeout is far better than nothing — the failure you're actually insuring against is not a bad answer, it's a blank screen.

You can see this system in action on Codelit. Generate a diagram and it'll route to the best available model automatically. On the Pro plan, you get whatever the current frontier models are for the complex stuff. On the free tier, you get the free model pool that handles most common architectures just fine.