Design an Agent Workflow Orchestrator: A Real Answer

Ask an engineer to design a URL shortener and you get a competent answer in forty minutes, because there are two hundred worked versions of that problem on the internet. Ask the same engineer to design the thing that actually runs an AI agent — the orchestrator underneath the demo — and the answer comes apart around minute twelve, usually on the sentence "and then it waits for someone to approve it."

That sentence is the whole problem. Everything before it is a request handler. Everything after it is a distributed system.

An agent workflow orchestrator

is a durable execution engine that drives an AI agent through a sequence of model calls and tool calls, checkpointing after every step so a run can pause for a human, survive a crash, and resume from exactly where it stopped.

I build Codelit, which has an agent workflow engine in it, so read the rest knowing I have a horse in the race. Where mine is narrower than the general design: every run is capped at twelve steps and a dollar of model cost, which is deliberately the wrong shape for a long-horizon research agent, and approvals go out by email today with Slack still on the list.

An agent workflow orchestrator has to do six things: accept a workflow definition, execute it step by step against real models and tools, pause on demand for a human, resume later from durable state, enforce hard budgets, and record enough of every run to debug it a week later. Two of those six shape the design; the other four fall out of them.

A run can pause for an unbounded length of time. An approval gate is not a slow dependency you survive with a longer timeout. It is a suspension that might last four minutes or four days, across a deploy, a process restart, a laptop closing. Accept that and request-scoped execution is dead — you are building durable state whether you wanted to or not.

Steps have side effects in systems you do not own. You cannot un-send a Slack message or un-merge a pull request. That forces idempotency keys, risk classification per tool, and a read-only default. It is also why the gate exists, so the two constraints are really one constraint wearing two hats.

Everything else — which model, which framework, JSON parsing versus a native tool-use API — is swappable. Watching engineers learn system design on a canvas, the failure mode is always the same: they know what every component does and freeze on which one the problem needs. This one needs durability and a gate.

You size an agent orchestrator on parked runs, not on requests per second. At ten thousand runs a day averaging eight steps each, you get eighty thousand steps a day — under one per second averaged, maybe ten at peak. That throughput makes no architectural demand on anything. The number that does is how many runs are suspended waiting for a person.

• Tokens. At 4,000 input and 1,000 output tokens per step, that is 400 million tokens a day. This is the number your finance conversation is actually about, and it scales with prompt growth, not with users.
• Storage. A step record holding the assembled prompt, tool arguments, and completion runs maybe 20 KB — 1.6 GB a day, about 150 GB at a 90-day window, which is how long Step Functions keeps a closed execution's history by default.
• Parked runs. If a fifth of runs hit an approval gate and the median human takes four hours, Little's Law gives you 2,000 runs a day times a sixth of a day: roughly 330 runs suspended at any moment.

Three hundred and thirty parked runs is nothing to hold in a database and impossible to hold in a process. That is the entire argument for durable execution over a long-lived worker with a timer, and it is worth deriving on the whiteboard rather than asserting.

A run moves through seven states, and the one people leave out is the one that matters: a run that completed six of nine steps and stopped is not a failed run, it is a stopped run with six durable results. Model that separately or you throw away work every time something goes wrong.

The schema follows from the table. A runs row holds identity, the pinned workflow version, the budget envelope, and current state. A run_steps row holds one record per attempted step with its own state, attempt count, and idempotency key. A run_events table is append-only, unique on (run_id, seq), and is the only thing permitted to be the source of truth — the other two are projections you can rebuild.

Pin the workflow version at run start. A run that began under version 3 finishes under version 3, or a mid-flight edit silently rewrites the meaning of every suspended run — and suspended runs are the ones you have most of.

You append an event to the run log after every completed step, and you keep the model call outside the code path that gets replayed, because replay assumes determinism and a model call is the least deterministic thing in your stack. Temporal's documentation says this directly: API calls, database queries, and LLM invocations belong in Activities, which execute outside the replay path.

That split is the most important structural idea here. Workflow code decides what happens next and must produce the same command sequence on every replay. Effects — the model call, the tool call, the Slack post — run once, get recorded, and are read back from the log rather than re-executed.

Then the log develops a size problem. Temporal limits Event History to 51,200 events or 50 MB and warns at 10,240 events or 10 MB. Step Functions caps a Standard execution at 25,000 history events, with a 256 KiB ceiling on the input or output of any single task or state, per the current AWS service quotas page. Those ceilings were designed for workflows whose payloads are identifiers. An agent step carries a four-thousand-token prompt and a thousand-token completion, and 256 KiB stops being generous fast.

The fix is boring and correct: blobs in object storage, content-addressed pointers in the log. History stays small enough to fold quickly, payloads stay as large as the model wants, and deduplication comes free because identical prompts hash to the same key. Secrets never enter the log at all — they resolve at dispatch from an encrypted vault, which is how Codelit's agent engine handles credentials, and the run log is the artifact you are most likely to hand someone while debugging.

One consequence gets missed most often: the context window is derived state, not stored state. On resume you do not restore a conversation object; you rebuild the prompt from the durable log under a declared compaction policy — last N steps verbatim, summarise the rest, always keep the original instruction and the approval decisions. Treat the conversation as authoritative and you will not be able to resume anything older than an hour.

Granularity is per step. Sub-step checkpointing buys nothing, because you cannot resume half a model call — the retry is always the whole call.

A tool call is safe to retry when the orchestrator — not the model — owns execution, validates the call against a declared schema, classifies it by risk, and attaches an idempotency key derived from the run identifier, the step sequence number, the tool name, and the canonicalised arguments. The model proposes. The orchestrator disposes.

Risk classification does the work, and there are only three classes. Read steps retry freely, in parallel, with backoff; this is the default and most steps should live here, which is why Codelit's live runs are grounded in read-only reads from GitHub, Jira, Linear, and Notion. Reversible write steps retry once with a key, and you register the compensating action before dispatch rather than after failure — if you cannot name the compensation, the step is not reversible and you have mislabelled it. Irreversible write steps are never automatically retried and always gated behind a person.

The class is yours to assign, not the tool's to declare. The MCP specification is unambiguous: clients must consider tool annotations untrusted unless they come from trusted servers, and the same spec says there should always be a human in the loop able to deny tool invocations. When I built ResolveMesh's read-only filter over discovered MCP tools, that was the founding assumption — a provider's description of its own tool is inventory to verify, never a safety guarantee to act on.

Then the deduplication window, which almost nobody sizes correctly. Stripe stores idempotency key results for at least 24 hours, prunes them after, and errors if you replay a key with different parameters. Good contract, not long enough for you: a gate that lands in an inbox on a Friday afternoon outlives every provider's dedupe window comfortably. Keep your own ledger, written before the call and updated after it.

Writing it before handles the nastiest ordering bug in this design — a tool call that succeeds while your orchestrator dies before recording the result. Those two writes cannot be atomic across a network boundary. What you can do is record intent first, so on resume a step sitting in dispatched with no result is one you must reconcile rather than one you are free to retry.

Fan-out gets one rule. Parallel steps need a barrier and a partial-failure policy declared at authoring time: fail fast, collect all, or proceed on quorum. Default to collect-all for reads, because a missing enrichment is usually survivable, and fail-fast for writes, because a half-applied change is not.

Every step type gets its own timeout and retry policy, because a classification call and a Jira ticket creation have nothing in common except that they travel over HTTP. A single global retry setting is the most common way to turn a twelve-step cap into a forty-step bill.

Two rules live under that table. Every step timeout must be shorter than the wall-clock budget it consumes, or one hung dependency eats a run's entire allowance without producing a single result. And every retry decrements the step budget, not just the dollar budget, because a flapping tool is the cheapest way to exhaust a run.

Budgets are admission control, not accounting. The scheduler checks the remaining allowance before dispatching a step, reserves that step's worst-case cost, executes, then reconciles against actual usage. Add costs up only afterwards and you have built a receipt, not a cap.

Four budgets, because each catches a different failure. Steps catch loops, and this is the one that actually stops a runaway agent. Tokens catch context growth, which creeps rather than spikes. Dollars catch expensive routing, since one frontier-model planning call can outweigh twenty classifications. Wall clock catches a dependency that is up but not answering.

Codelit's agent runs are capped at twelve steps and a dollar of model cost, and the pairing is the point: the step cap stops the loop, the dollar cap stops the single expensive step. It is also that engine's clearest limitation — twelve steps is a product decision aimed at people prototyping a workflow, not at a research agent that legitimately needs eighty. The budget model below still holds at that size. The numbers in it are mine, not yours.

Reserving output cost is easy — max_tokens times the output price is a ceiling you control. Input is harder, because the prompt grows as the run accumulates history. Count the assembled prompt before you dispatch it; you already have the string, and measuring after the response arrives is measuring the wrong thing at the wrong time.

Routing is the other lever, and a bigger one than people expect. Cheap models classify, stronger models plan. Codelit scores request complexity before routing and walks a fallback chain across eleven providers, so the same workflow costs wildly different amounts depending on how many of its steps are genuinely hard. Design the workflow so most of them are not.

You store what actually happened, at step granularity: the exact assembled prompt, the exact tool arguments, the exact response, the model and provider that answered, the token counts, the elapsed time. A non-deterministic system cannot be reasoned about from its source code, only from its trace.

That gives you three execution modes, and a good design ships all three:

This is the shape Codelit's agent engine ships in, and the part I am most confident is right. A dry run walks the whole flow step by step without calling a single real model or tool. A live run executes against real models with the gates in place and reports what the run cost against the estimate. Exporting a workflow produces a repo pack — runbook, approval policy, MCP config, model routing, a starter orchestrator, CI evals — with the best live run baked in as a fixture that replays on every commit.

The replay fixture is what I would argue hardest for in an interview. Evals over synthetic prompts tell you the agent is plausible. A replay of a run that genuinely worked tells you the change you just shipped did not break it, which is the question you actually have on a Tuesday afternoon.

One constraint on the trace: it holds whatever your tools read, so redact at write time. Redacting at read time means the unredacted version already exists somewhere, and somewhere is usually a log aggregator with a much longer retention policy than you intended.

Nine things break, and only two of them are the model being wrong. This is the table I would spend the last five minutes of a design interview on, because it is the one that shows you have run something rather than drawn it.

The injection row deserves its own sentence. Once an agent reads from the open web, or from a ticket anyone can file, every tool result is attacker-influenced input. The defence is not a better prompt. It is that the write path is gated by policy the model cannot argue with, which is why the gate is enforced in the orchestrator rather than requested in the system prompt.

A good interviewer adds two constraints in the last ten minutes: make it multi-tenant, and degrade a provider while runs are in flight. Both aim at the same soft spot — everything above quietly assumed one tenant's runs and one healthy model.

"Now make it multi-tenant." Four things bend. Secrets move to a per-tenant vault resolved at dispatch and never logged. The worker pool needs weighted fair queuing keyed by tenant, or one tenant with four hundred queued runs starves everyone else. Budgets grow a layer, because a per-run cap protects the tenant from itself while only a per-tenant spend and rate ceiling protects your provider account. And you need a bulkhead on in-flight steps per tenant, since a tenant whose tool endpoint has gone slow will otherwise occupy every worker with steps that are technically still healthy.

"Your primary provider has been degraded for two hours and you have runs mid-flight." The reflex answer is to fail over, and for a stateless step that is correct. For a run it is not. A plan produced by one model is not portable halfway through — swap between step four and step five and you get a run whose reasoning came from two models with no record of the seam.

So classify steps at authoring time as model-portable or model-pinned. Portable steps take the fallback chain immediately. Pinned steps pause the run into a resumable state with the degradation recorded as the reason, and a health check resumes them when the provider returns. A run that has spent eighty percent of its allowance should never restart from scratch on another model, because the durable log already paid for those steps. Resume beats restart, and the log is what makes resume possible. The other half of the answer is the health model — Codelit puts a recovered provider on probation with a fraction of traffic before restoring it, because a half-recovered provider will happily take a fresh wave of runs down with it.

The interview answer and the real one differ in exactly one place. In the interview, the approval gate is a safety valve you bolt on to satisfy the "but what if it does something bad" question. In a product, the gate is the feature — it is what makes an agent deployable against systems where being wrong is expensive, and every other decision here exists to support it. Durable state exists because gates suspend runs. Idempotency ledgers exist because suspension outlives dedupe windows. Partial-failure states exist because a gated run that stops is a run you want to keep.

Most writing about agents is about the model layer, which is why there is no good published answer to this question. The orchestrator is where the engineering lives, and it turns out to be a low-throughput, high-durability state machine with a fussy budget model and an unusually large trace — much closer to a payments workflow than to a chatbot.

If you would rather see one than read about one, Codelit's agent workflows will dry-run a whole flow before calling anything real. That first dry run is usually where people find out which of their steps were irreversible all along.