Rendering an LLM Response Before the JSON Closes

Constrained decoding solved one problem and created another. OpenAI's structured outputs launch post, published 6 August 2024, reports the then-new gpt-4o-2024-08-06 scoring 100% on their complex JSON-schema-following evaluation against under 40% for gpt-4-0613. The output became trustworthy. It also stayed completely unparseable until the final token, because JSON.parse on 90% of a document does not return 90% of an object. It throws.

So the default interface for schema-constrained generation has exactly two states: spinner, then everything.

A tolerant incremental parser

is a JSON reader that closes every string, array and object still open in a partial buffer before parsing a copy of it, so a half-written response yields a valid partial object instead of a syntax error.

Codelit — the architecture tool I build and run solo — streams its generated graph onto the React Flow canvas while the model is still writing it. Nodes appear as they close, edges join once both endpoints exist. This is the parser, the identity rules and the reconciler that make that safe.

Now the disclosure, and the part these posts usually skip. I build Codelit, and progressive rendering makes one specific thing about it worse. A product that shows a spinner and then one finished diagram can never show you a node that later disappears. Codelit can. If the Zod validation loop in the generation pipeline rejects a response and retries, the second answer may be a different architecture with different node ids, and the canvas has to swap underneath you. Two-state interfaces have no such failure mode. If your users read and act on the screen faster than your stream closes, the spinner is the better product and you can stop here.

Because JSON is only valid when it is closed, and constrained decoding constrains the finished document rather than each prefix of it. The sampler is masked away from tokens that would break the schema, which makes the last token a guarantee and every token before it a fragment.

The providers say so in their own documentation. Anthropic's streaming reference describes tool input arriving as content_block_delta events of type input_json_delta, each carrying a partial_json string, and states that the deltas are partial JSON strings whereas the final input is always an object. The suggested options are to accumulate the fragments and parse once the content block stops, or to do partial JSON parsing yourself.

Everything below is the second option.

They do not arrive at a steady rate, either. Your buffer sits still and then jumps, so whatever you build on top of it has to look alive during the still parts — a design problem long before it is a parsing one.

It repairs a copy of the buffer and parses that. On each chunk it scans for the last position where the document could legally stop, truncates a copy there, appends a closing brace or bracket for every container still open, and hands the result to JSON.parse. The accumulated buffer itself is never modified.

Two pieces of state make it small. The scanner tracks whether it is inside a string, and it tracks the difference between an object key and an object value — a closing quote after a colon ends a complete value and is a safe place to stop, a closing quote before one is a dangling key and is not. Miss that distinction and you will happily emit {"id": "gateway", "ty"}, which is not JSON.

type Frame = "object" | "array";

/**
 * Returns a parseable version of a truncated JSON document, or null when the
 * buffer has not yet reached a point that can be closed. The buffer itself is
 * never mutated — every chunk repairs a copy.
 */
export function closeJson(buffer: string): string | null {
  const stack: Frame[] = [];
  const awaitingValue: boolean[] = []; // parallel to stack; only meaningful for objects

  let inString = false;
  let escaped = false;
  let stringIsKey = false;

  let safeEnd = -1;
  let safeStack: Frame[] = [];

  // Record a position where the document could legally stop.
  const mark = (end: number) => {
    safeEnd = end;
    safeStack = [...stack];
  };

  for (let i = 0; i < buffer.length; i++) {
    const ch = buffer[i];

    if (inString) {
      if (escaped) escaped = false;
      else if (ch === "\\") escaped = true;
      else if (ch === '"') {
        inString = false;
        if (!stringIsKey) mark(i + 1); // a finished string value is a safe stop
      }
      continue;
    }

    switch (ch) {
      case '"':
        inString = true;
        stringIsKey =
          stack[stack.length - 1] === "object" && !awaitingValue[awaitingValue.length - 1];
        break;
      case ":":
        awaitingValue[awaitingValue.length - 1] = true;
        break;
      case ",":
        mark(i); // whatever preceded the comma was complete
        if (stack[stack.length - 1] === "object") awaitingValue[awaitingValue.length - 1] = false;
        break;
      case "{":
      case "[":
        stack.push(ch === "{" ? "object" : "array");
        awaitingValue.push(false);
        mark(i + 1); // an empty object or array is already valid
        break;
      case "}":
      case "]":
        stack.pop();
        awaitingValue.pop();
        mark(i + 1);
        break;
    }
  }

  if (safeEnd < 0) return null;

  let out = buffer.slice(0, safeEnd);
  for (let i = safeStack.length - 1; i >= 0; i--) {
    out += safeStack[i] === "object" ? "}" : "]";
  }
  return out;
}

safeEnd and safeStack always move together, which is the only subtle thing in there. The closers you append have to match the container stack as it stood at the truncation point, not as it stands at the end of the buffer.

Bare numbers and booleans land one token late, because the scanner only marks a safe stop after a closing quote, a comma, or a bracket. {"replicas": 3 rewinds to {} until the comma after the 3 arrives. That is a deliberate trade: the fields you want on screen early are strings, and refusing to guess where a number ends removes an entire class of bug for free.

When it satisfies the minimum shape your renderer needs, which is almost never the full schema. A Codelit node needs an id and a type to exist on the canvas. The label, the tier and the properties can all arrive later, and the node renders with a placeholder until they do.

That means two schemas, not one. The full contract still runs when the stream closes. A smaller drawable schema decides what gets admitted early.

import { z } from "zod";

// The contract the pipeline validates against once the stream closes.
const NodeSchema = z.object({
  id: z.string(),
  type: z.enum(["database", "service", "queue", "cache", "gateway", "client", "external"]),
  label: z.string(),
  tier: z.enum(["client", "edge", "application", "data", "external"]),
  properties: z.record(z.unknown()),
});

// The minimum shape the canvas needs to put something on screen.
const DrawableNode = NodeSchema.partial().required({ id: true, type: true });

export function drawableNodes(partial: unknown) {
  const nodes = (partial as { nodes?: unknown[] } | null)?.nodes ?? [];
  return nodes.flatMap((node) => {
    const parsed = DrawableNode.safeParse(node);
    return parsed.success ? [parsed.data] : [];
  });
}

Edges need a second rule: an edge is drawable only when both of its endpoints are already on the canvas. Streamed edges routinely reference nodes that have not arrived yet, so they go into a pending list that gets flushed every time a node lands. Without that, React Flow silently drops half your connections and you spend an afternoon blaming the layout engine.

Vercel's AI SDK documents both ends of this trade-off on one page. Partial outputs streamed from streamText "cannot be validated against your provided schema, as incomplete data may not yet conform to the expected structure", while in array output mode each element emitted by elementStream is "complete and validated against your element schema". Fast and unvalidated, or safe and late. A drawable schema is the deliberate middle: complete enough to render, incomplete on purpose.

Because React destroys and rebuilds a component when its key changes, and a streamed item changes constantly. Key a node on its label, its array index, or a hash of its data and every fragment that touches it unmounts the old component and mounts a new one.

What that costs, in order of how quickly users notice: the enter animation replays, so the canvas strobes. Selection is lost. An in-progress drag is dropped. Any local state inside the node — an open properties popover, a focused rename field — is gone. Use the id the model assigned and nothing else.

Memoisation makes this sharper rather than softer. Codelit's custom node components are wrapped in React.memo with a comparator, because React Flow hands every visible node a new position object on every frame of a pan. A streaming canvas applies the same pressure from the other direction: data changes many times per node per second, and the comparator is the only thing standing between you and a full re-render of the graph on every chunk.

Then there is layout, which is the part that actually decides whether streaming feels good. Dagre positions a graph, not a node, so adding one node moves everything.

The third is the only one I would ship, with pinning. It also leans on something the layout has to be anyway: deterministic. If the settle pass moved boxes to different places on identical input, the transition at the end would read as the tool changing its mind rather than tidying up.

You reconcile by id, and you never repaint. Diff the validated final graph against canvas state into three sets — add, update, withdraw — and apply them to the existing node array. Rebuilding a fresh array from the final object remounts every node on screen at the precise moment the user is first allowed to touch them.

function reconcile(current: Node[], authoritative: Node[]) {
  const byId = new Map(current.map((node) => [node.id, node]));

  const next = authoritative.map((incoming) => {
    const existing = byId.get(incoming.id);
    byId.delete(incoming.id);
    if (!existing) return incoming; // arrived only in the final object

    return {
      ...existing, // position, selection, and anything the user already dragged
      data: { ...existing.data, ...incoming.data, state: "settled" as const },
    };
  });

  // Whatever is left in byId was drawn optimistically and is not in the final
  // graph. Do not drop it here.
  return { next, withdrawn: [...byId.values()] };
}

Spreading existing first is the whole point. Position, selection and user edits survive; only the fields the model actually sent get overwritten. It is the same instinct behind returning a diff instead of a new diagram when someone follows up with "add caching" — the canvas belongs to the user, and the model is allowed to amend it, not replace it.

The uncomfortable case is a wholesale replacement, and it is not rare. If validation fails and the retry produces a different architecture, or the router falls through to another provider, the ids will not match at all and every optimistically drawn node lands in the withdrawn set. The reconciler handles it correctly and it still looks like the tool blinked. This is the honest cost of drawing early, and no amount of animation makes it disappear.

Assume it will. A stream ends in four different ways and only one of them is success, so the terminal event is the only thing that entitles you to call a result finished.

Two of those are documented behaviour rather than my guesswork. OpenAI's structured outputs guide notes that a response can fail to match the schema "if for example you reach a max tokens limit and the response is incomplete". Anthropic's streaming reference states that the API may occasionally send errors in the event stream, giving overloaded_error — which would be an HTTP 529 outside a streaming context — as its example.

That second one deserves attention if you route across providers. The router's rule is that a 200 from an AI provider is not a success signal; the streaming corollary is that a stream which stopped is not a stream that finished. An error arriving inside a 200 response is a case a naive fetch wrapper will never see.

Two phases, never instantly. Mark the node withdrawn, let an exit transition of a couple of hundred milliseconds run while the rest of the graph holds still, and drop it from the store when the transition ends. A node that blinks out of existence reads as a bug. A node that fades while nothing else moves reads as a correction.

setNodes([
  ...next,
  ...withdrawn.map((node) => ({
    ...node,
    data: { ...node.data, state: "withdrawn" as const },
  })),
]);
// The node component renders state === "withdrawn" with an exit transition and
// fires a callback on transitionend. Only then does the node leave the store.

Out of that falls the design rule that governs the whole feature: only render optimistically what you are willing to withdraw. Anything a user can immediately act on — click into, rename, export to a Terraform file — waits for the terminal event, because withdrawing something after they have started using it is worse than never showing it. Streaming is a decision you make per surface, not per product.

One cost that is documented and structural, and one you have to measure on your own schema. OpenAI's structured outputs guide states that the first request you make with any schema carries additional latency while the API processes it, and that subsequent requests with the same schema do not. The bill therefore arrives per schema version, on the first call after a deploy, not on every request — which is a very different thing to budget for.

Anthropic's side of it is a shape cost rather than a time cost. Because models currently emit one complete key and value property at a time, tool input arrives in bursts with gaps between them, and the documentation says as much. There is also a documented lever: fine-grained tool streaming, enabled per tool, streams tool input JSON without server-side buffering for lower latency.

None of that changes the arithmetic much. Generation runs about four seconds for a fifteen to twenty node diagram, and constrained decoding is not what dominates it.

It does not make anything faster, and speed is not what it buys. Jakob Nielsen's response time limits, published in 1993 and still posted on the Nielsen Norman Group site, put 1.0 second as "the limit for the user's flow of thought to stay uninterrupted" and 10 seconds as "the limit for keeping the user's attention focused on the dialogue".

Four seconds sits between them. That band is the worst place to be: too long to feel like a response, too short to justify a progress bar and a cancel button. It is also the band where the only lever left is what happens during the wait, because the generation itself is as fast as the model is going to be.

Streaming spends that band differently. Instead of one four-second gap with a spinner in it, the first node lands early and the rest arrive as a sequence of separate events. Nielsen's guidance for delays past one second is to indicate that the computer is working on the problem, and a node appearing is a considerably better indication than a spinner, because it is the answer arriving rather than a promise that an answer exists.

What it costs is honesty about state. The canvas now has a stretch where it is showing you something that might still change, and you have to signal that stretch rather than hide it.

I would write the reconciler before the parser. The parser is the fun part, it is sixty lines, and it is the bit everyone publishes. The reconciler is what decides whether the feature is trustworthy, and it is the thing you cannot retrofit, because by then every component in the tree has assumed items only ever get added. I would also make streaming a per-surface switch from the first commit, so "should this one draw early?" is a config change rather than an argument.

I said in the post about the Mermaid importer that I would not hand-write that parser again, and I stand by it — a Mermaid grammar grows without limit. This one I would hand-write every time. A brace counter with a key-versus-value flag has nowhere to grow to, no dependency to track, and no version that will surprise you. It is a rare case where the small hand-rolled thing is the correct long-term answer.

Go generate something at codelit.io and watch the canvas fill in. If a node ever appears and then vanishes on you, now you know exactly which branch you found.