Mo Sharif
← ~/blog

Learning System Design: Why Building Beats Reading

I've written 500+ blog posts on system design: URL shorteners, rate limiters, chat systems, payment processors, notification pipelines. Interview-prep staples plus the real-world architectures nobody writes about — event sourcing, feature flag platforms, multi-tenant SaaS backends.

Then I watched a lot of engineers use Codelit to learn, build, and break those architectures, and the place they got stuck was not the place I had been writing for.

System design judgment is the ability to decide which architectural component a specific constraint calls for, as distinct from knowing what each component does.

People don't struggle with concepts. They struggle with when to use what.

Engineers who can define every component still fail at system design because naming a component and choosing one are separate skills: the knowledge is complete, but the judgment about which constraint calls for which piece was never trained.

Ask any mid-level engineer what a load balancer does. They'll nail it. Ask what a message queue is for. Perfect answer. Cache? Database replication? CDN? They know the individual pieces cold.

Now ask them: "Your e-commerce checkout is timing out under load. Where do you add a cache, and what caching strategy do you use?"

Silence. Or worse, a wrong answer that sounds right.

Knowing what a message queue does is different from knowing that your notification system needs one because synchronous email sending is blocking your API response time. That second skill, the architectural intuition, doesn't come from reading. It comes from building.

And that realization changed everything about how I built Codelit.

Reading an architecture write-up builds recognition, while rebuilding that same architecture on a canvas builds judgment — the difference between reciting what a component does and knowing which trade-off applied where, and why.

I wrote those blog posts with the best intentions. Detailed explanations, diagrams, trade-off analysis. And they're useful. People tell me they are. But the numbers behind Codelit's growth told a different story about how people actually learn.

The people who only read a post could recite the concepts back to me. The people who read a post and then rebuilt the architecture on the canvas could reason with those concepts later. Reading produced recognition. Building produced judgment.

This isn't surprising if you think about it. You don't learn to cook by reading recipes. You learn by burning things and figuring out why.

How you engage with an architectureWhat it buildsWhere it stops
Read the write-upRecognition — you can name every componentYou still can't say which constraint asked for it
Rebuild it from memoryRecall, plus an exact list of what you forgotWhat you produce is still only the happy path
Modify itTrade-off sense — you feel what each swap costsFailure stays hypothetical until something fails
Break it deliberatelyFailure intuition — single points of failure jump outNeeds a working design first, so it can't be step one

So I leaned into it. Codelit lets you describe a system in plain English and see it generated as an interactive canvas you can drag, rewire and re-layout. The AI gives you a starting point. Your job is to understand it, question it, and improve it. That's where learning happens.

The 55 pre-built architecture templates beat both the blog posts and the AI generator on usage because a finished, correct system is the only thing you can meaningfully test yourself against, and testing yourself is what people were actually after.

There's Uber's ride-matching system. Netflix's content delivery pipeline. Stripe's payment processing flow. Twitter's timeline architecture. Slack's real-time messaging. WhatsApp's message delivery. Plus dozens of less glamorous but equally important systems: inventory management, appointment scheduling, IoT data ingestion. I wrote up the patterns that repeated across all 55 separately.

I expected people to use them as reference material. What they actually do is more interesting.

The most common workflow: load a template, study it for five minutes, close it, then try to recreate it from scratch using the AI generator. It's like looking at a solved puzzle, closing the box, and trying to solve it yourself. The gaps between what you remember and what you forgot are exactly the things you need to learn.

Then they modify them. They swap out components. They add features. They intentionally break them with chaos mode, then fix what broke. That cycle, generate, modify, break, fix, understand, is the feedback loop that actually teaches system design.

Beginners get system design wrong in five predictable ways: they choose infrastructure before requirements, default to microservices, draw arrows without deciding what flows through them, copy FAANG architectures at hobby scale, and never draw the failure path.

Starting with infrastructure instead of requirements. Beginners jump straight to "I need Kubernetes and Kafka and Redis" before they've asked "what does this system actually need to do?" I see architectures with a service mesh for an app that serves 50 requests per minute.

Treating every problem like it needs microservices. The default mental model for junior engineers in 2026 is microservices. They'll design a TODO app with six services, an API gateway, and a message broker. Sometimes a monolith is the right call. Usually, actually, for early-stage products.

Ignoring data flow. Beginners draw boxes and connect them with arrows, but don't think about what's flowing through those arrows. Is it synchronous? Async? How much data? What happens when the connection fails? The arrows matter more than the boxes.

Cargo-culting from FAANG. "Netflix uses this, so I should too." Netflix serves 250 million subscribers. Your side project has 12 users and a dog. Architecture should match your actual scale, not your aspirational scale.

No failure modes. The happy path is always drawn. The "what happens when this database goes down" path is never drawn. Resilience is an afterthought, if it's a thought at all.

Senior engineers make different mistakes, not fewer: they optimize for scale that hasn't arrived, leave inherited technology choices unquestioned, treat complexity as a proxy for quality, and find out what an architecture costs only after it's designed.

This one surprised me more than the beginner list did.

Over-optimizing for problems they don't have yet. I see senior engineers spend weeks designing a multi-region active-active database setup for an app that's deployed in one region with 500 daily active users. Premature optimization is an architecture problem long before it's a code problem.

Not questioning inherited architecture. "We use Kafka because we've always used Kafka." Maybe you started with Kafka when you had genuine event streaming needs. Now you're using it as a simple message queue, and SQS would cost 80% less and require zero operational overhead.

Conflating complexity with quality. More boxes on the diagram doesn't mean better architecture. Some of the best systems I've seen are embarrassingly simple. A monolith with good module boundaries, a single PostgreSQL database with proper indexing, and a CDN. That's it. And it handles millions of requests.

Ignoring cost until it's too late. Senior engineers design beautiful architectures and then discover they cost $15,000/month to run. The cost estimator that prices an architecture before you build it exists because I got tired of seeing this.

Every one of these is a judgment failure rather than a knowledge failure, which is why the fix is a question rather than a fact:

MistakeWho it hits hardestThe question that prevents it
Infrastructure before requirementsBeginnersWhat does this system have to do before it has to scale?
Microservices as the defaultBeginnersWhat actually breaks if this ships as one deployable?
Arrows without data flowBeginnersIs this call synchronous, how much data moves, and what happens when it fails?
Copying FAANG at hobby scaleBeginnersWhat is my traffic today, not my aspirational traffic?
Only the happy path drawnBothWhat is the blast radius when each box dies, and who notices first?
Optimizing for scale that hasn't arrivedSeniorsWhat evidence do I have that this load is coming?
Inherited choices never re-examinedSeniorsIf we were picking this today, would we pick it again?
Complexity mistaken for qualitySeniorsWhich box could I delete and still meet the requirement?
Cost discovered after designSeniorsWhat is the monthly bill for this diagram?

Engineers who use chaos mode to break architectures learn faster than engineers who only build them, because watching a design you just made collapse produces a reaction that no written warning about retry logic can match.

The pattern is clear. The people who regularly break and fix architectures on the canvas spot single points of failure much faster in new designs. They instinctively add retry mechanisms, circuit breakers, and fallback strategies. They think about failure modes from the start.

The mechanism is emotional, not intellectual. When you watch your carefully designed system collapse because you forgot a retry mechanism, you feel it. That "oh no, everything's broken" moment is a better teacher than any blog post. Including mine. The mechanics of it — how a single node failure cascades across the diagram — turned out to be the easy part to build.

I started pairing chaos mode failures with specific blog posts. "Your system failed because of X. Here's a post about how to prevent that." The click-through rate on those contextual recommendations is 4x higher than generic "related posts" suggestions. Context makes content stick.

AI generation removes the cold-start problem in system design: you used to need architectural knowledge to draw a meaningful diagram and diagrams to build architectural knowledge, which is circular, frustrating, and where a lot of people quit.

You say "design me a chat application" and you get a reasonable starting point even if you've never designed a distributed system. The pipeline that turns that sentence into a diagram is a story of its own, but what matters for learning is that you suddenly have something concrete to look at, question, and improve.

I watch users do this constantly. They generate a basic architecture, then start asking questions. "Why is there a WebSocket server separate from the API?" "What happens if the message queue goes down?" "Why do we need both a cache and a database?" Each question leads to deeper understanding. Each answer leads to another question.

The AI doesn't always generate the perfect architecture. And honestly? That's a feature. When the AI skips rate limiting, and a user asks "what happens if someone hammers this endpoint?", that's a teaching moment no pre-written content could have timed as well.

System design is a practice rather than a subject, which means you learn it by building, breaking, iterating, and arguing with other engineers about whether you really need that Kafka cluster — not by studying it.

The blog posts I wrote are still valuable. But they got far more useful the moment people could load them next to a live canvas and apply what they read on the spot.

The 55 pre-built architectures work less like reference material than like conversation starters. "Here's how Netflix does it. How would you do it differently? Why?" That question, in front of an interactive diagram you can actually modify, is worth more than a shelf of distributed systems textbooks.

I didn't set out to build a learning platform. I set out to build an architecture tool. The teaching was a side effect of making the feedback loop fast enough that people stopped being afraid to break things. My job is mostly to stay out of the way and keep the canvas fast.

Check out the templates and break something. That is the part that sticks.

Questions people actually ask

Why can engineers explain system design concepts but still design bad systems?
Explaining a component and choosing one are different skills. Most engineers can define a load balancer, a cache and a message queue accurately. What they cannot do is look at a checkout endpoint timing out under load and say which of those three the constraint actually calls for. That second skill is judgment, and it comes from building and breaking real designs rather than reading about them.
What is the fastest way to learn system design?
Build the systems, then break them. Load a reference architecture, study it for a few minutes, close it, and rebuild it from memory. The gaps between what you remembered and what you forgot are precisely the parts you have not learned yet. Then introduce failures deliberately and repair what falls over. Reading produces recognition, rebuilding produces recall, and breaking produces the failure intuition that shows up in real designs.
What mistakes do beginners make in system design?
Five recur constantly. Beginners pick infrastructure before requirements, reaching for Kubernetes and Kafka before asking what the system has to do. They default to microservices where a monolith would serve better. They draw boxes and arrows without deciding what flows through the arrows or what happens when a connection fails. They copy architectures from companies operating at a scale they will never reach. And they draw only the happy path.
Do senior engineers make system design mistakes too?
Yes, just different ones. Senior engineers over-engineer for scale that has not arrived, designing multi-region active-active databases for single-region apps. They inherit technology choices and never re-examine them, running Kafka as a plain message queue long after the streaming requirement disappeared. They mistake complexity for quality. And they find out what an architecture costs to run only after it is designed, which is the expensive moment to find out.
Does AI make it easier to learn system design?
It removes the cold-start problem. Learning system design used to be circular. You needed architectural knowledge to draw a meaningful diagram, and you needed to draw diagrams to build architectural knowledge. Generating a first draft from a plain-English description gives a beginner something concrete to interrogate, and the questions are where the learning happens. The generated design does not have to be correct. Its gaps are the lesson.
Why does breaking an architecture teach more than building one?
Because failure is felt rather than read. When a design you just made collapses because you forgot a retry, the reaction sticks in a way that a written warning about retries does not. Engineers who regularly break and repair architectures start spotting single points of failure in unfamiliar designs, and they add circuit breakers, retries and fallbacks by reflex instead of by checklist.