Chaos Engineering for Architecture Diagrams (On Purpose)

A user emailed me and asked: "Can I see what happens if my database goes down?"

I read that email three times. It was such an obvious feature that I felt stupid for not thinking of it. Architecture diagrams show you how a system is connected. They don't show you what happens when part of it breaks, which is arguably the more important half. The connections don't matter if you don't understand the failure modes.

Chaos mode

is a simulation layer for an architecture diagram that fails a component you pick, walks the dependency graph outward from it, and marks every downstream service as failed or degraded.

So I built it. You click a component, hit the "fail" button, and watch cascading failures propagate through your architecture in real time. Red pulses spread from node to node. Services degrade or die depending on their dependency type. At the end you can see exactly what survived and what didn't. It's the feature users talk about more than anything else.

Architecture diagrams draw connections and stop there. They show you that a service talks to a database. They never show you that the service dies when the database does. Every real system has failure modes, and a static diagram is silent about all of them.

A database going down doesn't just affect the database. It cascades to every service that depends on it. A queue backing up doesn't kill the producers, but the consumers starve. A CDN outage doesn't crash your backend, but your users see broken images.

You can stare at a diagram and mentally trace all of that, but you'll miss things. Humans are bad at reasoning about cascading effects in complex graphs. We forget about the monitoring service that depends on the logging service that depends on the database that just died.

Going through 55 real-world system architectures for the template library, most of the interesting decisions in them turned out to be failure-mode decisions — a replica here, a dead letter queue there — and none of that was legible from the boxes and arrows.

Chaos mode runs a breadth-first search from the failed node, following edges outward and computing a failure state for every node it reaches. It is not a plain BFS: the edge type decides whether the node on the other end dies, degrades, or shrugs it off.

type DependencyType = "hard" | "soft" | "async";

interface PropagationResult {
  affectedNodes: Map<string, FailureState>;
  propagationPath: PropagationStep[];
}

function propagateFailure(failedNodeId: string, nodes: Node[], edges: Edge[]): PropagationResult {
  const affected = new Map<string, FailureState>();
  const path: PropagationStep[] = [];
  const queue: { nodeId: string; depth: number }[] = [];

  // Mark the initial node as failed
  affected.set(failedNodeId, { status: "failed", cause: "direct" });
  queue.push({ nodeId: failedNodeId, depth: 0 });

  while (queue.length > 0) {
    const { nodeId, depth } = queue.shift()!;

    // Find all nodes that depend on the failed node
    const dependents = edges
      .filter((e) => e.source === nodeId)
      .map((e) => ({
        targetId: e.target,
        depType: (e.data?.dependencyType as DependencyType) || "hard",
      }));

    for (const { targetId, depType } of dependents) {
      if (affected.has(targetId)) continue; // already processed

      const failureState = calculateFailureState(targetId, depType, edges, affected);

      if (failureState.status !== "healthy") {
        affected.set(targetId, failureState);
        path.push({
          from: nodeId,
          to: targetId,
          depth: depth + 1,
          type: depType,
        });
        queue.push({ nodeId: targetId, depth: depth + 1 });
      }
    }
  }

  return { affectedNodes: affected, propagationPath: path };
}

The depth counter isn't decoration. It's what the animation uses to decide which layer of the cascade renders when.

A hard dependency kills the downstream service, a soft dependency degrades it, and an async dependency lets it keep running while the damage lands somewhere else later. Chaos mode reads the type off the edge and picks the failure state from it.

Async is the one people get wrong. Your order service publishes events to a message queue, the queue dies, and orders still process. Nothing looks broken. What's broken is the analytics service that stopped updating an hour ago, and nobody finds out until someone opens a dashboard on Monday.

function calculateFailureState(
  nodeId: string,
  incomingDepType: DependencyType,
  edges: Edge[],
  currentlyAffected: Map<string, FailureState>
): FailureState {
  // Check for redundancy: does this node have multiple sources?
  const allIncomingEdges = edges.filter((e) => e.target === nodeId);
  const healthySources = allIncomingEdges.filter((e) => !currentlyAffected.has(e.source));

  // If there are healthy replicas/alternatives, the node survives
  if (healthySources.length > 0 && incomingDepType === "hard") {
    return { status: "degraded", cause: "partial-dependency-loss" };
  }

  switch (incomingDepType) {
    case "hard":
      return { status: "failed", cause: "hard-dependency" };
    case "soft":
      return { status: "degraded", cause: "soft-dependency" };
    case "async":
      return { status: "degraded", cause: "async-dependency" };
  }
}

Chaos mode checks whether an affected service still has a healthy incoming connection before declaring it dead. A database with three replicas can lose one and survive, and a simulation that reddens the whole diagram anyway is a scaremonger with a colour palette.

If your API server connects to both postgres-primary and postgres-replica-1, and only the primary fails, the API server degrades instead of dying. The canvas shows it in yellow (degraded) instead of red (failed).

That creates a feedback loop. You run chaos mode, see that your payment service has a single point of failure on one database, and think "I should add a replica." You add the replica in the diagram, run chaos mode again, and see that the failure now degrades the service instead of killing it. The diagram becomes a design tool, not just a documentation tool.

Because a cascade that renders instantly isn't a cascade, it's a diagram that turned red. Chaos mode delays each layer by 300ms so you can see the order services fell over in, and which ones were hit directly rather than transitively.

All of this rides on the same interactive React Flow canvas the diagrams already use, so failure state could stay a pure styling layer: extra class names on existing nodes and edges, no second renderer.

@keyframes failurePulse {
  0% {
    box-shadow: 0 0 0 0 rgba(239, 68, 68, 0.7);
  }
  50% {
    box-shadow: 0 0 0 12px rgba(239, 68, 68, 0);
  }
  100% {
    box-shadow: 0 0 0 0 rgba(239, 68, 68, 0);
  }
}

.node-failed {
  border-color: #ef4444;
  background-color: #fef2f2;
  animation: failurePulse 1.5s ease-in-out infinite;
}

.node-degraded {
  border-color: #f59e0b;
  background-color: #fffbeb;
  animation: failurePulse 2s ease-in-out infinite;
}

.edge-propagating {
  stroke: #ef4444;
  stroke-dasharray: 6 3;
  animation: flowAnimation 0.5s linear infinite;
}

Yes. When the propagation finishes, chaos mode sends the failure data to the model — which node failed, which nodes were affected and how, the dependency types involved, whether redundancy existed — and asks for mitigations written against that specific diagram.

It runs through the same prompt-to-diagram pipeline as generation, so it inherits the multi-provider fallback. A suggestion panel that returns nothing because one vendor is having a bad afternoon is worse than none.

"The payment-service has a hard dependency on a single postgres-primary instance with no replica. Adding a read replica and configuring automatic failover would prevent complete service failure when the primary goes down."

Or:

"The notification-service depends on the email-queue via an async dependency. Consider adding a dead letter queue to capture failed events and a retry mechanism to reprocess them when the queue recovers."

These aren't generic advice. They name the nodes in front of you. Users have told me this is the most useful part of chaos mode: the breakage gets the attention, the fix list is what people act on.

Four decisions turned chaos mode from a demo into something people use twice: keep failure state out of the saved document, let failures compound, stay deterministic, keep the traversal cheap.

Failure state is transient. Chaos mode doesn't modify the actual diagram data. It applies a visual overlay. When you exit chaos mode, everything goes back to normal. This was a deliberate choice. Users should feel free to break things without worrying about messing up their diagram.

You can fail multiple nodes. Click one node to fail it, see the cascade, then click another. The cascades compound. This is how you test scenarios like "what if my database AND my cache go down simultaneously?"

The propagation is deterministic. Same diagram, same failed node, same result every time. This is important because users compare before and after when they modify their architecture. If the results were random, the comparison would be meaningless.

Performance scales fine. The BFS traversal is O(V + E) where V is nodes and E is edges. For a 50-node diagram (which is large by Codelit standards), the computation is under 1ms. The staggered animation takes much longer than the algorithm.

The reaction has been out of proportion to the three weeks the feature took to build. I don't usually quote user feedback because it feels self-serving, but these were genuinely surprising:

"This is the coolest thing I've ever seen in a diagramming tool."

"I used this to prep for a system design interview and the interviewer was impressed that I could talk about failure modes so specifically."

That one matches what I've seen about how engineers actually learn system design: failure modes are the line between memorising a reference architecture and understanding one.

"Found a single point of failure in our production architecture that none of us had noticed. We added redundancy the next week."

That last one is the one that makes it all worth it. Someone used a feature I built to improve a real production system.

Chaos mode ships on every diagram in Codelit. Generate or load an architecture, click the chaos mode button, pick a component, and watch it break. It also runs on the pre-built templates — the Uber architecture is a good one to chaos-test because it has enough components that the cascading failures get interesting. The chaos mode feature page shows it in action if you want to watch before trying.

It started as a "wouldn't it be cool if" idea in one user's email and turned into the feature that defines the product. Sometimes the best features aren't the ones you plan. They're the ones your users describe and you're smart enough to listen.