System Design

System Design

System design is the discipline of making architectural decisions that allow a product to handle growing load without rewriting it from scratch. The best-designed systems are not the most complex — they are the ones that are simple enough to operate and extend as requirements change.

The Scaling Journey

Most products follow a predictable scaling path:

Phase 1 (0–1,000 users): Single server, single database. All traffic hits one machine. Cost: $20–100/month. This is correct. Do not over-engineer early.

Phase 2 (1,000–10,000 users): Vertical scaling (larger server), add a read replica to offload analytical queries, add a CDN for static assets. Cost: $100–500/month.

Phase 3 (10,000–100,000 users): Horizontal scaling with a load balancer in front of multiple application servers. Add a caching layer (Redis). Move file storage to object storage (S3). Separate background jobs to dedicated workers.

Phase 4 (100,000+ users): Database sharding or read replicas for specific tables, microservices for high-load components, message queues for async work, global CDN for content delivery.

The trap is designing for Phase 4 when you have 500 users. Netflix, Uber, and Amazon's architectures are the result of years of evolution under production traffic — not upfront design decisions.

Stateless Architecture

The prerequisite for horizontal scaling is stateless application servers. If your server stores session data in memory, every user must be routed to the same server on every request. With two servers, user A's session is on Server 1, so if Server 2 handles A's next request, it cannot find the session.

The fix: store state outside the servers. Sessions go in Redis. Uploaded files go in S3. Any state that must persist beyond a single request lives in an external data store, not in server memory.

Caching Strategy

Cache invalidation is famously one of the two hard problems in computer science. The strategy:

Cache what is expensive and frequently read: database query results, rendered HTML fragments, third-party API responses. The cache is a solved set of the most common queries so the database only handles the long tail.

Cache at the right layer:

• CDN caches static assets and full page HTML at the network edge (closest to users)
• Application cache (Redis) caches computed values shared across requests
• Database query cache (automatic in some databases) caches query plans and results

Cache invalidation strategies:

• TTL (Time-to-Live): cache expires after N seconds automatically. Simple but may serve stale data.
• Write-through: update the cache every time you update the source. No stale data, more write complexity.
• Event-driven: invalidate specific cache keys when the underlying data changes.

Async Processing with Queues

Synchronous operations that take more than ~200ms hurt user experience. If a user uploads a photo and must wait 8 seconds while your server resizes it to 5 dimensions, the UX is poor. The pattern: accept the upload immediately, return a response, and enqueue the resizing job for a background worker.

Message queues (BullMQ with Redis, AWS SQS) enable this decoupling. The API server is the producer; background workers are consumers. Workers can be scaled independently — if image processing is the bottleneck, add more worker instances without touching the API servers.

Queues also provide retry semantics: if a worker crashes mid-job, the job returns to the queue and another worker picks it up. This gives you at-least-once processing guarantees.

Microservices vs. Monolith

The default for new products should be a monolith — one codebase, one deployment, one database. Microservices introduce distributed systems complexity (network latency between services, distributed transactions, independent deployment pipelines) that is a maintenance burden for small teams.

The right time to extract a microservice: when a specific component has different scaling requirements than the rest of the system (e.g., your ML inference service needs GPUs while your CRUD API needs only CPU), or when different teams need to deploy independently.

Amazon famously started as a monolith. Their "two-pizza team" microservices architecture emerged after years of growth, not as an upfront design choice.

Monitoring and Observability

You cannot improve what you cannot see. Production monitoring requires three pillars:

• Metrics: numerical measurements over time (request rate, error rate, p50/p95/p99 latency, database connection count). Tool: Datadog, Grafana, or Vercel Analytics.
• Logs: structured text records of events (request logs, error logs, audit logs). Tool: Logtail, Papertrail, AWS CloudWatch.
• Traces: end-to-end tracking of a request through every component it touches, with timing for each step. Tool: Jaeger, Datadog APM. Traces tell you where in your stack a slow request is spending its time.