System design in five questions

The first time you try to study system design it feels like memorizing a dictionary. Load balancing, replication, sharding, the CAP theorem, message queues, rate limiting, circuit breakers, distributed tracing, CDNs. Each term has its own diagram and its own war stories, and it is easy to come away thinking the field is a pile of unrelated tricks you either happen to know or you do not.

It is not. After enough years building systems that had to survive real load, from a low-latency trading engine to analytics serving a couple hundred thousand users, I stopped seeing a glossary and started seeing five questions. Every concept in that pile is an answer to one of them. Learn the questions and the terms stop being trivia and start being options you reach for on purpose.

Here is the shape of a system, and where each question lives.

One: how do the parts talk to each other?

The moment a system is more than one process, you have to decide how the pieces communicate, and every choice is a trade. A load balancer sits in front and spreads requests across many copies of a service so no single one drowns. Between services, a plain REST call is simple and universally understood, while something like gRPC is faster and stricter and pays off for chatty internal traffic. And when work does not need to happen right now, a message queue lets one service drop a job and move on while another picks it up later.

That last one is the quiet workhorse. The instinct of a junior system is to do everything inside the request: take the order, charge the card, send the email, update the warehouse, all before responding. The architect’s move is to do the part the user is waiting on and hand the rest to a queue. The response gets faster and the slow steps get to fail and retry without the user ever seeing it.

Two: where does the data live when there is too much of it?

One database on one machine is fine until it is not. Two techniques push that wall back. Replication keeps multiple copies of the data, typically with one primary taking the writes and replicas serving reads, which both spreads read load and gives you a survivor if the primary dies. Sharding splits one enormous dataset across many machines by some key, so no single box has to hold all of it.

Both buy you room and both cost you simplicity. Replicas can lag behind the primary, so a read right after a write might see stale data. Sharding means a query that spans shards is suddenly a distributed problem. You take these on when the data genuinely outgrows one machine, not before, because each one adds a class of bug that a single database never had.

Three: how do reads get fast?

Most systems read far more than they write, and the same expensive answer gets computed over and over. A cache stores the hot results in fast memory, usually something like Redis, so the common reads never touch the database at all. It is the highest-leverage performance move there is, and it comes with the single hardest problem in the field: knowing when the cached copy is stale and throwing it away at the right moment. Cache the wrong thing for too long and you are confidently serving yesterday’s answer. Invalidate too eagerly and the cache stops helping. Getting that timing right is most of the work.

Four: what happens when something fails?

At small scale you assume things work. At real scale you assume they break, because at enough volume something always is. The patterns here are about failing gracefully instead of catastrophically. Rate limiting refuses excess traffic before it can take the whole system down. A circuit breaker notices that a downstream service is failing and stops hammering it, giving it room to recover instead of piling on. Retries with growing wait times handle the blips without turning a hiccup into a stampede. And distributed tracing lets you follow one request across all the services it touched, which is the only way to debug anything once a single call fans out across ten of them.

None of these make the system faster on a good day. They are the difference between a bad day being a slow response and a bad day being an outage.

Five: how does it grow?

Finally, the question every successful system eventually faces. You can scale vertically, giving one machine more CPU and memory, which is easy and hits a hard ceiling fast. Or you scale horizontally, adding more machines behind a load balancer, which has almost no ceiling but demands that your application hold no state on any single box, because the next request might land on a different one.

That statelessness requirement is why the earlier questions matter so much. You can only add machines freely if the session does not live in one server’s memory, if the data lives in a shared store, if the slow work is on a queue. Scaling is not a thing you bolt on at the end. It is the payoff for decisions you made much earlier.

The one idea underneath all five

Notice what every answer had in common: it was a trade. Caching buys speed and owes you staleness. Replication buys resilience and owes you lag. Horizontal scale buys headroom and owes you statelessness. Queues buy responsiveness and owe you eventual consistency. There is no move on this board that is free.

That is the actual skill, and it is the reason system design cannot be reduced to memorizing the glossary. The experienced engineer is not the one who can define sharding. It is the one who, faced with a real system, asks the five questions, and for each answer knows exactly what it is trading away and whether this particular system can afford the bill. The terms are just the menu. Knowing what you are paying is the job.