Chapter 4: Scaling - System Design Course: Complete Foundations for Interviews | PlayBook Library

SLIDE 48: Why Systems Fail Under Load

Section 5: Scaling

Every system works fine with 10 users. The challenge is making it work with 10 million. Scaling is the process of evolving a system’s architecture as traffic grows, and it follows a predictable progression. This section covers two things: the scaling strategies themselves (vertical, horizontal, caching, sharding), and the capacity estimation math that tells you when to apply each one.

What We’ll Cover:

Vertical vs horizontal scaling
The scaling progression from 1 user to 1 billion
Back-of-envelope estimation (QPS, storage, bandwidth)

What Happens When a System Can’t Handle Load:

Structure & Concept Flow

Stage 1: Slow responses

- Server CPU at 80%+, request queue building up

- P50 response time: 200ms → 2,000ms

- Users notice pages loading slowly

Stage 2: Timeouts and errors

- Server CPU at 95%+, memory pressure

- Some requests time out (504 Gateway Timeout)

- 5-10% of requests fail

Stage 3: Cascading failure

- Database connections exhausted

- Application servers crash and restart

- Load balancer health checks fail, traffic shifts to remaining servers

- Remaining servers now receive even more traffic, fail faster

- Entire system goes down

This progression can happen in minutes during a traffic spike.

The goal of scaling is to ensure you never reach Stage 1.

The Core Principle:

Scaling is not about building for 1 billion users on day one. It is about knowing what breaks next and having a plan for when it does. Over-engineering is as wasteful as under-engineering. The right architecture depends on your current scale and your expected growth rate.

SLIDE 49: Vertical Scaling (Scale Up)

Vertical Scaling

Vertical scaling means upgrading to a more powerful machine: more CPU, more RAM, more disk, faster network. The application code doesn’t change. You just run it on better hardware.

Example:

CODE

Current server (AWS m5.large):
- 2 vCPUs, 8 GB RAM, 500 GB SSD
- Handles: 500 requests/sec
- Cost: $70/month

Upgraded server (AWS m5.4xlarge):
- 16 vCPUs, 64 GB RAM, 1 TB SSD
- Handles: 4,000 requests/sec
- Cost: $560/month

8x the capacity for 8x the cost. Linear cost scaling.

Advantages:

Advantage	Explanation
Simple	No code changes, no architectural changes
No distributed systems complexity	Single machine, no network calls between servers
ACID preserved	Single database, transactions work naturally
Fast to implement	Resize the instance, restart, done

Limits:

CODE

The biggest cloud server available (as of 2024):

AWS u-24tb1.metal:
- 448 vCPUs
- 24,576 GB RAM (24 TB)
- Cost: ~$220,000/month

Even this machine has a ceiling:
- Single CPU has a clock speed limit
- Single disk has an IOPS limit
- Single network card has a bandwidth limit
- Single machine is a single point of failure

There is no server powerful enough to run YouTube or Instagram.
At some point, you MUST distribute across multiple machines.

When Vertical Scaling Makes Sense:

Small to medium traffic (under 10,000 requests/sec)
Database servers (often easier to scale up than shard)
Quick fix while you build a horizontal scaling solution
Applications that are hard to distribute (legacy monoliths)

SLIDE 50: Horizontal Scaling (Scale Out)

Horizontal Scaling

Horizontal scaling means adding more machines instead of upgrading existing ones. Traffic is distributed across all machines using a load balancer.

Example:

CODE

Vertical approach:
1 server (16 vCPU, 64 GB RAM) = 4,000 requests/sec
Cost: $560/month

Horizontal approach:
8 servers (2 vCPU, 8 GB RAM each) = 8 × 500 = 4,000 requests/sec
Cost: 8 × $70 = $560/month

Same capacity, same cost. But horizontal gives you:
- Fault tolerance (1 server dies, 7 still running)
- Linear scaling path (need more? add server #9, #10, #11...)
- No upper ceiling (keep adding machines indefinitely)

How It Works:

Requirements for Horizontal Scaling:

CODE

Your application MUST be stateless for horizontal scaling to work.

Stateless: Any server can handle any request.
- Session data stored in Redis (not in server memory)
- Authentication via JWT tokens (validated by any server)
- File uploads go to shared storage (S3, not local disk)

If your application stores anything in local memory or local disk
that another server would need, horizontal scaling breaks.
This is why Section 3 emphasized stateless servers.

SLIDE 50B: Vertical vs Horizontal Scaling: Comparison

Comparison:

Dimension	Vertical (Scale Up)	Horizontal (Scale Out)
Approach	Bigger machine	More machines
Complexity	Low (no code changes)	Higher (requires stateless design, load balancer)
Upper limit	Hardware ceiling (biggest available server)	No ceiling (add machines indefinitely)
Cost curve	Gets expensive fast (premium hardware pricing)	Linear (commodity hardware)
Fault tolerance	None (single point of failure)	Built-in (machines are redundant)
Downtime for scaling	Yes (must restart to resize)	No (add new server, start receiving traffic)
Database support	Works well (scale up the DB server)	Complex (requires sharding or read replicas)

In Practice, You Use Both:

CODE

Typical production setup:

Application servers: Horizontally scaled
- 10-1,000+ stateless servers behind a load balancer
- Auto-scaling adds/removes servers based on CPU usage

Database: Vertically scaled first, then horizontally
- Start with a large instance (vertical)
- Add read replicas when reads exceed capacity (horizontal)
- Shard when storage or write throughput hits limits (horizontal)

Cache (Redis): Horizontally scaled
- Redis cluster with multiple nodes
- Data partitioned across nodes

The real-world answer is never purely one or the other.

SLIDE 51: Scaling Stages 1-2 (0-10K Users)

Scaling Progression: Stage 1 and 2

Stage 1: Single Server (0-100 Users)

CODE

Architecture:
- Everything on one machine
- Application code + database + web server
- No redundancy, no caching, no CDN

Capacity: 100-500 requests/sec
Cost: $50-100/month
Bottleneck: CPU and RAM shared between app and database

Stage 2: Separate Database (100-10,000 Users)

CODE

What changed:
- Database moved to its own machine
- App server and DB server can be sized independently
- Database gets more RAM (for caching query results)
- App server gets more CPU (for processing requests)

Capacity: 1,000-5,000 requests/sec
Cost: $150-300/month (2 servers)
Bottleneck: Single app server (CPU limited)
Next step: Add a load balancer and more app servers

SLIDE 52: Scaling Stages 3-4 (10K-1M Users)

Stage 3: Load Balancer + Multiple Servers (10,000-100,000 Users)

CODE

What changed:
- Load balancer distributes traffic across app servers
- App servers are stateless (session data in external store)
- Can add/remove servers based on traffic
- One server failing doesn't take down the system

Capacity: 5,000-15,000 requests/sec
Cost: $500-1,500/month
Bottleneck: Single database handling all reads and writes
Next step: Database replication

Stage 4: Database Replication (100,000-1,000,000 Users)

CODE

What changed:
- Read replicas handle read traffic (which is usually 80-95% of queries)
- Primary handles writes only
- Read capacity multiplied by number of replicas

Capacity: 15,000-50,000 requests/sec
Cost: $2,000-5,000/month
Bottleneck: Slow reads for frequently accessed data (hitting database for every request)
Next step: Add caching layer

SLIDE 53: Scaling Stages 5-6 (1M-100M Users)

Stage 5: Caching Layer (1,000,000-10,000,000 Users)

CODE

What changed:
- Redis cache sits between app servers and database
- Hot data (popular profiles, trending posts) served from cache
- Cache hit: ~0.5ms response (from RAM)
- Cache miss: ~5-50ms response (from database)
- 80% cache hit rate means 80% fewer database queries

Capacity: 50,000-200,000 requests/sec
Cost: $5,000-15,000/month
Bottleneck: Static assets (images, CSS, JS) served from origin servers
Next step: CDN for static content

Stage 6: CDN + Object Storage (10,000,000-100,000,000 Users)

CODE

What changed:
- Images, videos, CSS, JS served from CDN edge servers worldwide
- Object storage (S3) for user-uploaded media
- CDN cache hit: ~30ms (from edge server near user)
- Origin no longer serves static content directly
- 80%+ of bandwidth offloaded to CDN

Capacity: 200,000-500,000 requests/sec
Cost: $15,000-100,000/month (CDN bandwidth is expensive)
Bottleneck: Single database primary can't handle write volume or storage growth
Next step: Database sharding

SLIDE 54: Scaling Stages 7-8 (100M-1B+ Users)

Stage 7: Database Sharding (100,000,000-1,000,000,000 Users)

CODE

What changed:
- Database split into 100-1,000+ shards
- Each shard holds a subset of users and their data
- Shard key: user_id % number_of_shards
- Each shard has its own primary + replicas
- Write throughput scales linearly with shard count

Capacity: 500,000-5,000,000 requests/sec
Cost: $100,000-1,000,000/month
Bottleneck: Monolith application is hard to scale, deploy, and maintain with large teams
Next step: Decompose into microservices

Stage 8: Microservices + Message Queues (1,000,000,000+ Users)

CODE

What changed:
- Monolith split into independent services (User, Post, Feed, Media, etc.)
- Each service scales independently
- Async processing via message queues (Kafka, SQS)
- Service mesh for inter-service communication
- Multiple teams deploy independently

Capacity: 5,000,000+ requests/sec
Cost: $1,000,000+/month
This is where YouTube, Instagram, Netflix, and Amazon operate.

Additional infrastructure at this stage:
- Service mesh (Envoy/Istio) for service discovery
- Distributed tracing (Jaeger, Datadog) for debugging
- Feature flags for gradual rollouts
- Circuit breakers for fault isolation
- Auto-scaling groups for each service independently

SLIDE 55: Complete Scaling Progression

All 8 Stages on One Diagram:

Stage	Architecture	Users	Requests/sec	Monthly Cost
1	Single server (app + DB)	0-100	100-500	$50-100
2	App server + separate DB	100-10K	1K-5K	$150-300
3	Load balancer + multiple app servers	10K-100K	5K-15K	$500-1,500
4	Database read replicas	100K-1M	15K-50K	$2K-5K
5	Redis caching layer	1M-10M	50K-200K	$5K-15K
6	CDN + object storage	10M-100M	200K-500K	$15K-100K
7	Database sharding	100M-1B	500K-5M	$100K-1M
8	Microservices + queues	1B+	5M+	$1M+

Key Observations:

Structure & Concept Flow

1. Each stage solves the bottleneck from the previous stage:

CPU → DB separation → Single server → Replicas → DB load → Cache →

Static assets → CDN → Storage/writes → Sharding → Team/deploy → Microservices

2. Cost grows roughly with user count:

$0.001/user/month at Stage 2

$0.01/user/month at Stage 5

$0.10/user/month at Stage 8

3. You don't need all 8 stages from day one.

Most startups live at Stage 3-4 for years.

Only a handful of companies ever reach Stage 7-8.

4. Each stage adds complexity.

Only move to the next stage when the current one breaks.

Premature optimization wastes engineering time and money.

Interview Tip:
When designing a system, ask the interviewer about scale. If they say “design for 10,000 users,” you don’t need sharding or microservices. If they say “design for 500 million users,” skip straight to Stage 6-8 and explain why. Matching architecture to scale demonstrates practical engineering judgment.

SLIDE 56: Back-of-Envelope Estimation

Capacity Estimation

Back-of-envelope estimation is the practice of calculating approximate infrastructure requirements using simple arithmetic. Interviewers use this to test whether candidates can reason about scale and translate user-facing requirements into infrastructure needs.

What to Estimate:

Metric	What It Tells You
QPS (Queries Per Second)	How many servers and database connections you need
Storage	How much disk space, what kind of database, when to shard
Bandwidth	CDN requirements, network costs, potential bottlenecks
Memory (Cache)	Redis cluster size, cache hit rate impact

Ground Rules:

CODE

1. Round aggressively. Precision doesn't matter, order of magnitude does.
   86,400 seconds/day ≈ 100,000 (acceptable rounding)

2. Show your work. The calculation process matters more than the answer.

3. State assumptions explicitly.
   "I'll assume 20% of users are active daily" is better than guessing silently.

4. Use powers of 2 or 10 for speed:
   1 KB   = 1,000 bytes    (10^3)
   1 MB   = 1,000,000 bytes (10^6)
   1 GB   = 1,000,000,000 bytes (10^9)
   1 TB   = 1,000,000,000,000 bytes (10^12)
   1 PB   = 1,000,000,000,000,000 bytes (10^15)

SLIDE 57: Estimating QPS

QPS Estimation

Formula:

CODE

Read QPS = (DAU × reads per user per day) / seconds per day
Write QPS = (DAU × writes per user per day) / seconds per day
Peak QPS = Average QPS × peak multiplier (typically 2-5x)

Worked Example: Instagram

CODE

Given:
- DAU: 500 million
- Posts viewed per user per day: 20
- Posts created per user per day: 0.2 (not everyone posts daily)

Read QPS:
= 500,000,000 × 20 / 86,400
= 10,000,000,000 / 100,000  (rounding 86,400 to 100,000)
= 100,000 QPS

Write QPS:
= 500,000,000 × 0.2 / 86,400
= 100,000,000 / 100,000
= 1,000 QPS

Peak QPS (assume 3x average):
- Peak read:  300,000 QPS
- Peak write: 3,000 QPS

Read:Write ratio = 100,000 : 1,000 = 100:1
This is a heavily read-skewed system.

What This Tells You:

CODE

Read QPS of 100,000:
- Single server handles ~5,000 QPS
- Need: 100,000 / 5,000 = at least 20 app servers
- With 3x headroom for peak: 60 app servers

Write QPS of 1,000:
- Single database handles ~10,000 writes/sec
- 1,000 writes/sec fits on a single primary (no sharding needed for writes alone)
- But storage growth will eventually force sharding

Read:Write ratio of 100:1:
- Invest heavily in read optimization (caching, read replicas, CDN)
- Write path can be simpler

SLIDE 58: Estimating Storage

Storage Estimation

Formula:

CODE

Daily storage = (new records per day) × (average record size)
Annual storage = Daily storage × 365

Worked Example: Instagram

CODE

Photo storage:
- 100 million posts/day
- Average photo size after compression: 2 MB
- Daily: 100,000,000 × 2 MB = 200 TB/day
- Annual: 200 TB × 365 = 73,000 TB = 73 PB/year

Metadata storage:
- Post metadata: ~1 KB per post
- Daily: 100,000,000 × 1 KB = 100 GB/day
- Annual: 100 GB × 365 = 36.5 TB/year

Comparison:
- Media (photos): 73 PB/year    (99.95%)
- Metadata (DB):  36.5 TB/year  (0.05%)

Key insight: Media storage completely dominates.
Architecture must optimize for media delivery (CDN + object storage).
Database storage for metadata is negligible by comparison.

What This Tells You:

CODE

73 PB/year of photos:
- Cannot store in a database (use object storage: S3, Azure Blob)
- Must use CDN for delivery (origin can't handle the egress bandwidth)
- Need lifecycle policies (move old photos to cheaper storage tiers)
- Cost at S3 standard pricing: 73,000 TB × $23/TB/month = $1.68M/month

36.5 TB/year of metadata:
- Fits on a handful of sharded PostgreSQL instances
- Each shard holds ~36 GB (with 1,000 shards)
- Comfortable within PostgreSQL's limits

SLIDE 59: Estimating Bandwidth

Bandwidth Estimation

Formula:

CODE

Ingress (upload) bandwidth = (writes per second) × (average write size)
Egress (download) bandwidth = (reads per second) × (average read size)

Worked Example: Instagram

CODE

Ingress (photo uploads):
= 1,000 writes/sec × 2 MB per photo
= 2,000 MB/sec
= 2 GB/sec

Egress (photo views):
= 100,000 reads/sec × 2 MB per photo
= 200,000 MB/sec
= 200 GB/sec

Bandwidth ratio:
Egress: 200 GB/sec (100x ingress)
Ingress: 2 GB/sec

This 100:1 egress-to-ingress ratio makes CDN non-negotiable.
No origin server cluster can sustain 200 GB/sec of outbound traffic.
CDN edge servers handle 80%+ of this egress.

What This Tells You:

CODE

Without CDN:
- Need origin bandwidth: 200 GB/sec
- Network cost: 200 GB/sec × $0.09/GB = $18/sec = $46M/month

With CDN (80% cache hit rate):
- Origin bandwidth: 40 GB/sec (80% served from CDN edge)
- CDN cost: much cheaper than origin bandwidth
- Total cost: significantly lower

CDN isn't just a performance optimization. At this scale,
it's a cost requirement. The math makes it unavoidable.

SLIDE 60: Worked Example: Chat Application

Complete Estimation: WhatsApp-Scale Chat System

CODE

Given:
- MAU: 2 billion
- DAU: 1 billion (50% daily engagement)
- Messages sent per user per day: 40
- Average message size: 100 bytes (text only)
- Group size average: 5 members (each message delivered to 5 people)

QPS:

CODE

Write QPS (messages sent):
= 1,000,000,000 × 40 / 86,400
= 40,000,000,000 / 100,000
= 400,000 messages/sec (sent)

Read QPS (messages delivered):
= 400,000 × 5 (average group size)
= 2,000,000 messages/sec (delivered)

Peak (3x): 1.2M writes/sec, 6M reads/sec

Key insight: This is a write-heavy system (unlike Instagram).
Write optimization is critical: Cassandra or similar write-optimized database.

Storage:

CODE

Daily message storage:
= 1 billion users × 40 messages × 100 bytes
= 4,000,000,000,000 bytes/day
= 4 TB/day

Annual: 4 TB × 365 = 1.46 PB/year

With media messages (photos, voice notes):
Assume 5% of messages have media, average 500 KB
= 2,000,000,000 media messages/day × 500 KB
= 1,000 TB/day = 1 PB/day (!)

Media storage dominates again: 365 PB/year for media vs 1.46 PB/year for text.

Bandwidth:

CODE

Text bandwidth:
= 2,000,000 messages/sec × 100 bytes
= 200 MB/sec (trivial)

Media bandwidth:
= 100,000 media messages/sec × 500 KB
= 50 GB/sec (significant, CDN needed)

Architecture Implications:

Metric	Value	Implication
Write QPS	400K/sec	Need write-optimized DB (Cassandra, not PostgreSQL)
Read QPS	2M/sec	Need aggressive caching + read replicas
Text storage	1.46 PB/year	Sharded database, TTL for old messages
Media storage	365 PB/year	Object storage (S3) + CDN
Media bandwidth	50 GB/sec	CDN required

SLIDE 61: Capacity Estimation: Common Mistakes

Common Mistakes in Capacity Estimation

CODE

Mistake 1: Forgetting to distinguish read vs write
"QPS is 100,000."
Problem: Read QPS and write QPS have completely different implications.
100,000 reads/sec needs caching and replicas.
100,000 writes/sec needs sharding and write-optimized databases.
Always state both separately.

Mistake 2: Not accounting for peak traffic
"Average QPS is 100,000, so we need servers for 100,000."
Problem: Traffic is not evenly distributed. Peak hours can be 3-5x average.
If you provision for average, the system goes down during peak.
Always calculate: peak QPS = average QPS × 3 (minimum).

Mistake 3: Precision over magnitude
"QPS = 10,000,000,000 / 86,400 = 115,740.74..."
Problem: You're spending interview time on long division.
Correct approach: 10B / 100K ≈ 100K QPS. Close enough.
Interviewers want to see you reason about scale, not do exact math.

Mistake 4: Ignoring metadata vs media storage
"We need 73 PB for the database."
Problem: 73 PB is media (object storage), not database.
Database stores metadata (~36 TB/year for Instagram).
Mixing these up leads to completely wrong infrastructure decisions.

Mistake 5: Hand-waving with "we'll scale horizontally"
"How will you handle 500M users?" "We'll just scale horizontally."
Problem: This is not an answer. Which component scales? How many servers?
What's the shard key? How many shards?
Always follow up with specific numbers.

Interview Tip:
Spend 3-5 minutes on capacity estimation in an interview. Write the numbers on the whiteboard as you calculate. The interviewer is watching your reasoning process, not checking your arithmetic. Round aggressively, state assumptions clearly, and draw concrete conclusions (“this means we need ~20 app servers and CDN is mandatory”).

SLIDE 62: Section Summary and Interview Tips

Section 5 Summary: Scaling

Concepts Covered:

Concept	One-Line Summary
Vertical scaling	Bigger machine (simple, has a ceiling)
Horizontal scaling	More machines (requires stateless design, no ceiling)
Scaling stages	Predictable progression from single server to microservices
Back-of-envelope estimation	Approximate QPS, storage, bandwidth from user requirements
Read vs write QPS	Different ratios require different optimization strategies
Peak vs average	Always provision for peak (3-5x average)

The Scaling Decision Tree:

Structure & Concept Flow

Problem: "System is slow / can't handle traffic"

Is it a single server?

→ Separate database from app server (Stage 2)

Is the app server overloaded?

→ Add load balancer + more app servers (Stage 3)

Is the database overloaded with reads?

→ Add read replicas (Stage 4)

Are reads still slow for popular data?

→ Add Redis caching layer (Stage 5)

Are static assets slow to deliver globally?

→ Add CDN (Stage 6)

Is the database too large or write throughput too high?

→ Shard the database (Stage 7)

Is the monolith hard to scale, deploy, or maintain?

→ Decompose into microservices (Stage 8)

Key Numbers to Remember:

CODE

Seconds in a day: ~100,000 (actual: 86,400)
Single web server: 1,000-5,000 QPS
Single database: ~10,000 QPS, ~10-16 TB practical limit
Redis: 100,000-500,000 ops/sec per node
CDN cache hit rate target: 80%+
Peak multiplier: 3-5x average
Instagram scale: 100K read QPS, 1K write QPS, 73 PB media/year

How This Connects to Other Sections:

CODE

Section 4 (Databases):  Replication and sharding are scaling techniques for the data layer
Section 6 (Caching):    Stage 5 of the scaling progression
Section 7 (Load Balancing): Stage 3 of the scaling progression
Section 8 (CDN):        Stage 6 of the scaling progression
Section 9 (Queues):     Part of Stage 8 (async communication between microservices)
Section 13 (Design Exercise): You'll apply estimation to a URL shortener design