Multi-Provider AI with Auto-Fallback

If you're building an AI product on a single provider, you're building on a single point of failure. I learned this the hard way when OpenAI had a 4-hour outage in early 2025 and Codelit was completely dead for the duration.

Never again.

Codelit now routes across 11 providers with automatic fallback. The result: 99.7% uptime over the last 6 months, including during multiple provider outages.

Requests are routed based on a priority hierarchy. If the top provider fails or times out, the system immediately tries the next one:

1. GPT-5 (OpenAI direct) is the best quality for architecture generation. Default for Pro users.
2. Claude Opus (Anthropic direct) is excellent at understanding complex system descriptions. Close second.
3. Gemini 2.5 (Google direct) is good quality with fast response times.
4. GPT-5 via OpenRouter is the same model on a different API endpoint. It catches OpenAI API issues that aren't model-level outages.
5. Claude via OpenRouter runs the same fallback logic.
6. Grok, Mistral, and the OpenRouter free pool are lighter models for when premium providers are all down.
7. DeepSeek or Groq (direct) are the last resort. Cheaper, but very reliable.

Free tier users start at position 3 or lower. Pro users get the full hierarchy starting from GPT-5. This is one of the main reasons to upgrade. The quality difference between the premium models and the lighter ones is noticeable, especially for complex architectures.

Not every request needs GPT-5. A simple "three-tier web app" diagram doesn't require the same model that "distributed event-driven microservices architecture with CQRS and event sourcing" does.

The system estimates prompt complexity before routing:

• Simple (fewer than 5 expected components): route to faster, cheaper models
• Medium (5-15 expected components): route to mid-tier models
• Complex (15+ components, specific technology requirements, iteration on existing diagram): route to premium models

This keeps costs manageable. If every request hit the premium model, the API costs would eat the entire margin on the $5/month Pro plan. Complexity-based routing means the expensive model only handles requests that actually need it.

Different models return slightly different JSON structures. GPT-5 might nest groups inside a groups array. Claude might use clusters. Gemini might inline group membership on each node.

The normalization layer sits between the AI response and the canvas renderer. It transforms every provider's output into Codelit's canonical schema:

interface NormalizedResponse {
  nodes: CanvasNode[];
  edges: CanvasEdge[];
  groups: CanvasGroup[];
  metadata: {
    provider: string;
    model: string;
    latencyMs: number;
    tokensUsed: number;
  };
}

This means the canvas doesn't know or care which AI generated the diagram. The user experience is identical regardless of which provider handled the request.

Running 11 AI providers isn't cheap. Here's how I keep costs sane:

Caching. Identical prompts (which happen more often than you'd think, since lots of people type "microservices architecture") return cached results. Cache hit rate is about 12%.

Token budgets. Each model call has a max token limit tuned to the complexity tier. Simple diagrams get a 1,000-token budget. Complex ones get 4,000. This prevents runaway costs from verbose responses.

Provider pricing arbitrage. OpenRouter sometimes offers the same model at a lower per-token price than the direct API. The routing layer checks current pricing and prefers the cheaper path when quality is equivalent.

Batched operations. When a user iterates ("add caching to this diagram"), the system sends only the delta (the existing diagram structure plus the modification request) rather than regenerating from scratch. This cuts token usage by 40 to 60% on iterations.

Monthly AI API spend is about $800-1,200 depending on usage volume. At current subscriber counts, that's sustainable with healthy margin.

The system doesn't wait for a full timeout to declare a provider failed. It uses:

• Response time monitoring. If a provider's p95 latency exceeds 15 seconds, it gets deprioritized
• Error rate tracking. More than 3 errors in a 5-minute window triggers automatic fallback
• Health checks. Periodic lightweight requests to verify each provider is responding

When a provider recovers, it's gradually reintroduced, first handling 10% of requests, then 50%, then back to full priority. This prevents a flood of requests from hitting a provider that just came back online and might be unstable.

• 99.7% uptime over the last 6 months
• Average generation time: 4.2 seconds (across all providers)
• Fallback events per month: ~15-20 (usually brief provider hiccups)
• Zero complete outages since implementing multi-provider (previously had 3 in 6 months)

Check out the features page and pricing to see how this translates to the product tiers.