Agent Orchestrator Design: Durable Runs and Safe Tools

The difficult part of an agent workflow is often one ordinary sentence: "Wait for someone to approve this."

Now the run has to survive a closed tab, a deploy, and a person who comes back on Monday. A longer HTTP timeout will not solve that.

Building Codelit pushed me to separate the impressive part of an agent demo from the part I would trust with a real workflow. This is the design I use to reason about that boundary, not a claim that every capability below is implemented in Codelit.

An orchestrator owns execution policy: which step may run, with which credentials, under which budget, and what happens after it stops. The model can propose actions, but it must not grant itself permissions or decide that an expired approval counts as consent.

I start with six responsibilities: durable state, tool validation, scheduling, approvals, budgets, and a trace. Provider choice comes after those. This is the same distinction I make when teaching system design: choose components from the constraints, not from the diagram you want to draw.

Size an orchestrator around concurrent model calls, suspended runs, and retained payloads, not only incoming requests per second. Slow dependencies and long approval waits can dominate resource use even when the average request rate looks small.

For an interview exercise, assume 10,000 runs a day and eight steps per run. That is 80,000 steps a day, under one per second on average. It says little about burst capacity or provider quotas.

If 20% of runs wait for approval and the mean wait is four hours, Little's Law gives roughly 333 suspended runs. Those figures are assumptions, not Codelit measurements. The architectural point is that suspended work belongs in durable storage; it should not depend on keeping a worker alive.

A useful run model distinguishes active work, human waits, resumable stops, and terminal outcomes. "Failed" alone is too coarse: a run with six completed steps should retain those results even when step seven cannot continue.

Keep the workflow version pinned to the run. Store step attempts separately from logical steps, so a retry does not overwrite its predecessor. Append-only events plus queryable projections are one design; transactional state transitions with an audit log can also work. Event sourcing is a choice, not a prerequisite for durability.

Deterministic replay restores control flow from recorded results; it must not casually repeat model calls or external writes. A recorded completion can be reused, but an attempt without a recorded result remains uncertain and may need another call or destination-side reconciliation.

Temporal separates Workflow replay from Activities, where external work belongs. Its Activity guidance recommends idempotent Activities because retries can execute work again. Durable execution does not make a network side effect exactly once.

Keep large prompts and responses in access-controlled object storage when history size requires it. Store pointers and integrity hashes in the run record, with explicit retention and tenant isolation. Redact secrets before persistence; resolving credentials at dispatch does not prevent a tool response from leaking one.

Rebuild model context from durable results under a declared compaction policy. Preserve the task, constraints, approval decisions, and references to omitted material. Compaction is lossy, so test resumed runs as a distinct path.

A tool call is retry-safe only when repeating the operation cannot cause an unacceptable second effect. A stable idempotency key helps when the destination honors it; a local ledger records uncertainty but cannot make two systems commit atomically.

Record intent, dispatch the call, then record the outcome. If the process dies after dispatch, leave the attempt in an explicit unknown state. Query the destination by operation ID if possible. Otherwise stop and ask for reconciliation.

Stripe may remove idempotency keys once they are at least 24 hours old. Other providers have different contracts. Set ledger retention from your workflow's lifetime and audit requirements, not a guessed universal window.

Tool descriptions are not permission grants. The MCP tools specification warns clients not to trust annotations from untrusted servers. That is also the starting point for ResolveMesh's capability filtering.

Budget checks belong in the scheduler, before work is admitted. Per-attempt timeouts belong at the call boundary. Together they prevent one slow dependency, retry loop, or parallel fan-out from silently consuming the whole run allowance.

Track model tokens, model cost, tool cost, attempts, and active execution time. Approval waiting time deserves its own expiry rather than sharing a short network timeout.

A reservation needs a defensible upper bound. Include assembled input, maximum output, provider-specific billable tokens, and any tool fees. If a service exposes no useful ceiling, use a conservative allowance and do not advertise an exact hard dollar cap.

Provider fallback must consume the same reservation and deadline. It is not a fresh budget every time another model gets a turn.

A dry run should prove the behavior of the simulation you wrote: reachable branches, required approvals, and budget rules under supplied fixtures. Live model quality, OAuth authorization, and actual external delivery remain separate acceptance tests.

A replay that injects yesterday's answer cannot evaluate today's prompt. Use fresh evaluations for that. Keep tests honest about which boundary they cross.

Review the cases where local state and external reality can diverge: a successful write with no recorded result, a revoked permission during a pause, a stale approval, and a canceled run with work still in flight. Those are product states, not just exceptions.

An approval should expire to hold. A changed argument should require a new approval. Cancellation should prevent new dispatch immediately while the trace separately records what happened to outstanding calls.

Provider changes need a compatibility decision too. Some steps are portable; others depend on provider-specific tool results or conversation state. Pause those steps when compatibility is uncertain instead of assuming a second model is interchangeable.

The design test I care about is simple: can the user tell what happened, what did not happen, and what needs permission next? If the trace cannot answer those questions after a crash, the orchestrator is not finished.