Chaos Mode: Finding Failure Paths Before Deployment

"What happens if this database goes down?"

That question was more useful than another request for a diagram shape. It turned Codelit's canvas from a picture of the happy path into a place to discuss what the system depends on.

I built it to make failure paths visible before someone has to discover them during an incident. The boundary matters: it reasons about the architecture you drew, not the infrastructure you deployed.

A diagram can expose declared dependencies and make potential failure paths easier to inspect. It cannot know hidden coupling, runtime load, network policy, or whether a failover mechanism actually works unless those facts are represented and tested elsewhere.

A database outage might stop checkout while leaving browsing available. A cache outage might increase database load until the database fails too. Both are richer behaviors than "the API turns red."

Working through architecture templates made this clear: the interesting decisions were often about what could fail independently. A useful simulation should make those decisions discussable instead of implying certainty from a color.

Failure propagation should follow dependency direction and evaluate the affected operation's declared requirements. Build an adjacency index, compute resulting states, and revisit dependents when an upstream state worsens. A first-visit-only traversal can miss a later loss of an alternative dependency.

Direction needs a documented convention. A request-flow arrow from API to database points toward the dependency, while failure propagates back toward the API. Do not assume the visual arrow already means "provider to dependent."

For a simple reachability graph, indexed breadth-first traversal can be linear in nodes plus edges. A richer model with alternatives, thresholds, and repeated state evaluation needs its own complexity analysis. Filtering the full edge list at every node is not the same algorithm.

Dependency types are shorthand for a behavior contract, not predictions inferred from service names. Specify which operation needs the dependency, whether a fallback exists, and how long any buffer or degraded path remains useful under the modeled workload.

A cache is only a soft dependency if the application can bypass it and the backing service can handle the load. A queue failure can block producers immediately if publishing is required to acknowledge a request.

Those details make the simulation useful. Without them, "async means degraded" is just a default color.

Redundancy exists when another resource can provide the same required capability under the failure being modeled. The alternative needs compatible behavior, sufficient capacity, and a defined switch-over path. An unrelated healthy edge does not satisfy that requirement.

The naive rule I would avoid is "if any incoming connection is healthy, mark the node degraded." An API can have a healthy cache and still be unable to write orders because its database is down.

Model alternatives as groups: one of these providers is required, or a stated quorum must remain. Keep read and write capabilities distinct. A database read replica may preserve reads without preserving writes.

That gives system design practice a better question than "did the boxes stay green?": what assumption changed, and how would you verify it?

Multiple failures should be evaluated together from the same scenario state, with a deterministic rule for how dependent states worsen. Otherwise the result can depend on which node the user clicked first rather than the architecture itself.

Useful fixtures include a diamond-shaped dependency, a cycle, two alternative providers failing in sequence, and an unrelated healthy dependency. Run each fixture with different input ordering.

Keep a reason alongside every status: failed because a required write capability disappeared, degraded because reads fell back to a slower source, or uncertain because the graph lacks enough information. "Unknown" is more useful than an invented success.

The interface should show the selected failure, the affected operations, and the dependency path that produced each status. Color can reinforce that explanation, but text and accessible labels must carry the meaning without requiring motion or color perception.

A staggered animation can teach traversal order. It is presentation time, not a prediction that an outage will propagate at the same speed. Provide reduced-motion behavior and a way to inspect the settled result immediately.

The React Flow canvas can display this as an overlay on existing nodes and edges. Keep the simulation state separate from the saved graph so leaving the mode does not rewrite the design.

AI mitigations should identify the assumption they address, the proposed change, and the test needed to verify it. Naming the user's nodes makes advice specific, but specificity alone does not make the advice correct.

Instead of "add a read replica and the service survives," a useful suggestion is: "This operation requires database writes. Decide whether a standby can be promoted, define the routing change, and test writes during failover."

Instead of "a dead-letter queue fixes a broker outage," ask where an event is durably stored before the broker accepts it. A dead-letter queue handles particular processing failures; it does not automatically capture a message that never reached the broker.

Suggestions can use the generation pipeline and its provider fallback, but they should remain reviewable proposals, not automatic changes.

Simulation stops at the model's assumptions. Chaos engineering introduces controlled failures into a running system and compares observed behavior with a hypothesis. A design simulation can help choose the experiment; it cannot substitute for the experiment's results.

The Principles of Chaos Engineering describe experiments around measurable steady-state behavior and controlled variables. Live testing also needs authorization, a bounded blast radius, stop conditions, and recovery preparation.

That is the payoff I want from chaos mode: not confidence because a diagram stayed green, but a sharper list of assumptions to test before users depend on them.