A model outage should not force me to redesign the product. But adding a second API key does not make an AI workflow reliable either.
In Codelit, I separate the provider-facing code from the architecture the user edits. That boundary lets routing change without making every canvas component understand a different API.
The boundary that matters
The provider returns a candidate result. The application owns whether that result is usable.
For an architecture diagram, that means checking node identity, supported types, edge references, and required fields before treating the output as the new state. A successful HTTP response is only the first check.
| Boundary | Responsibility |
|---|---|
| Routing | Select an available, eligible model path |
| Provider adapter | Translate request and response formats |
| Validation | Reject malformed or incompatible output |
| Product state | Preserve the user's work when an attempt fails |
| User feedback | Explain the outcome and the next useful action |
What fallback can and cannot do
Codelit's provider code includes alternate model paths and surfaces fallback feedback. The available path depends on the model, configured credentials, plan, and failure involved. There is no single fixed provider ranking that accurately describes every request.
Changing the endpoint can recover from some failures. It cannot guarantee that the underlying model is healthy, that a quota is available, or that a second model supports the same behavior. Two endpoints can also depend on the same upstream service.
The full routing article explains the design trade-offs and failure cases.
Cost is part of recovery
A retry is more work, not a free second chance. It can consume another request, repeat input tokens, and leave the outcome of the first attempt uncertain.
I evaluate routing in terms of a useful completed result, not the cheapest nominal token rate. That means accounting for invalid output, retries, response time, and the cost of recovery. The AI estimation worksheet makes those assumptions explicit.
The engineering lesson
The architecture keeps provider changes behind a stable product boundary. The canvas can keep the user's work intact while the request layer handles an unavailable or incompatible model.
The useful evidence is a bounded failure path: preserve existing state, validate the replacement result, make the provider change visible, and stop when no acceptable path remains.
That is the same principle behind durable agent orchestration. Reliability comes from defined behavior when work stops, not from a longer provider list.