Chapter 6: Load Balancing - System Design Course: Complete Foundations for Interviews | PlayBook Library

SLIDE 77: The Need for Load Distribution

Section 7: Load Balancing

In Section 5, we established that horizontal scaling requires distributing traffic across multiple servers. A load balancer is the component that makes this possible. It sits between clients and your server fleet, deciding which server handles each incoming request.

What We’ll Cover:

What a load balancer does and where it sits
Load balancing algorithms (round robin, least connections, IP hash)
Layer 4 vs Layer 7 load balancing
Health checks and failure detection
Load balancer redundancy
Global server load balancing (GSLB)

The Problem:

CODE

Without load balancer:
- Client must know the IP address of a specific server
- If that server is down, client gets an error
- If that server is overloaded, client gets slow responses
- No way to add or remove servers without changing client configuration

With load balancer:
- Client knows one address (the load balancer)
- Load balancer picks a healthy, available server
- Servers can be added/removed without clients knowing
- Traffic is distributed evenly across all servers

SLIDE 78: What a Load Balancer Does

Load Balancer Responsibilities

A load balancer performs several functions beyond just distributing traffic.

Function	Description
Traffic distribution	Spread requests across multiple backend servers
Health monitoring	Detect unhealthy servers, stop sending traffic to them
TLS termination	Decrypt HTTPS at the load balancer, forward HTTP to backends
Session persistence	Optionally route same user to same server (sticky sessions)
Connection management	Maintain persistent connections to backends, reducing handshake overhead
DDoS absorption	Absorb connection floods before they reach application servers

How It Works:

CODE

From the client's perspective:
- Client connects to lb.youtube.com (203.0.113.1)
- Client has no idea how many backend servers exist
- Client doesn't know which server handled their request

From the server's perspective:
- Servers have private IPs (10.0.1.x), not exposed to the internet
- Servers only receive traffic from the load balancer
- Servers can be added or removed without downtime

Common Load Balancer Technologies:

Technology	Type	Used By
Nginx	Software (open source)	Most common, general purpose
HAProxy	Software (open source)	High-performance TCP/HTTP
AWS ALB/NLB	Managed service	AWS-native applications
Azure Load Balancer	Managed service	Azure-native applications
Google Cloud LB	Managed service	GCP-native applications
F5 BIG-IP	Hardware/software	Enterprise, on-premise

SLIDE 79: Load Balancer Placement in Architecture

Where Load Balancers Sit

A production architecture typically has load balancers at multiple layers.

Three Layers of Load Balancing:

Layer	Purpose	Technology
Global (DNS-based)	Route users to nearest data center	Route 53, Cloudflare DNS, Azure Traffic Manager
Regional (L7 HTTP)	Distribute traffic across app servers within a region	Nginx, ALB, HAProxy
Internal (service-to-service)	Route between microservices within the data center	Envoy, internal NLB, service mesh

In a system design interview, you typically draw one load balancer between clients and application servers. If the interviewer asks about global distribution or microservices communication, mention the additional layers.

SLIDE 80: Round Robin and Weighted Round Robin

Load Balancing Algorithms

Round Robin:

The simplest algorithm. Requests are distributed to servers sequentially, one after another, cycling back to the first server after the last.

Structure & Concept Flow

Servers: A, B, C

Request 1 → Server A

Request 2 → Server B

Request 3 → Server C

Request 4 → Server A (cycle restarts)

Request 5 → Server B

Request 6 → Server C

...

Each server gets exactly 1/3 of the traffic.

When Round Robin Works:

All servers have identical hardware specs
All requests take roughly the same time to process
No session affinity required

When Round Robin Fails:

CODE

Problem: Servers are not equal.

Server A: 16 vCPUs, 64 GB RAM (powerful)
Server B: 4 vCPUs, 16 GB RAM  (weak)
Server C: 4 vCPUs, 16 GB RAM  (weak)

Round robin gives each server 33% of traffic.
Server A is underutilized, Servers B and C are overloaded.

Weighted Round Robin:

Assigns a weight to each server proportional to its capacity. Higher weight = more traffic.

Structure & Concept Flow

Server A: weight 4 (gets 4 out of every 6 requests)

Server B: weight 1 (gets 1 out of every 6 requests)

Server C: weight 1 (gets 1 out of every 6 requests)

Request 1 → A

Request 2 → A

Request 3 → A

Request 4 → A

Request 5 → B

Request 6 → C

Request 7 → A (cycle restarts)

...

Server A handles 67% of traffic, matching its 4x capacity advantage.

SLIDE 81: Least Connections and IP Hash

Least Connections:

Routes each new request to the server with the fewest active connections. This handles the case where some requests take much longer than others.

Structure & Concept Flow

Server A: 12 active connections

Server B: 3 active connections

Server C: 8 active connections

New request → Server B (fewest connections)

Why this is better than round robin for variable-length requests:

Round robin scenario:

- Requests 1-3 arrive. One per server. Equal.

- Request 1 (to Server A): image upload, takes 10 seconds

- Request 2 (to Server B): API call, takes 50ms

- Request 3 (to Server C): API call, takes 50ms

- Request 4 arrives: Round robin sends to Server A

But Server A is still busy with the upload!

Server B and C are idle.

Least connections scenario:

- Request 4 arrives: Server B has 0 active, Server A has 1

→ Sends to Server B. Better distribution.

IP Hash:

Routes requests from the same client IP to the same server every time. Computed by hashing the client’s IP address.

Structure & Concept Flow

server_index = hash(client_ip) % number_of_servers

Client 192.168.1.1 → hash → always Server A

Client 192.168.1.2 → hash → always Server C

Client 192.168.1.3 → hash → always Server B

Every request from the same IP goes to the same server.

When to Use IP Hash:

CODE

Use case: Server-side caching
- Server A caches data for its assigned clients
- If a client always goes to Server A, the cache hit rate is high
- If the client bounces between servers, each server has a cold cache

Use case: Stateful applications (legacy)
- Application stores session in local memory (not Redis)
- Client must go to the same server to find their session
- Note: Stateless design with external session store is the better approach

Problem: Uneven distribution
- Some IP ranges produce more traffic than others
- If a corporate office (one IP) sends 10,000 requests,
  one server gets all of them

SLIDE 81B: Load Balancing Algorithms: Comparison

Algorithm Comparison:

Algorithm	Distribution	Complexity	Best For
Round Robin	Equal	Lowest	Identical servers, uniform request cost
Weighted Round Robin	Proportional to weight	Low	Mixed server capacities
Least Connections	Dynamic (based on load)	Medium	Variable request duration (uploads, downloads)
IP Hash	Deterministic (by client)	Low	Session affinity, server-side caching
Least Response Time	Dynamic (fastest server)	High	When server performance varies
Random	Statistically equal	Lowest	Large server fleets where simplicity matters

Decision Framework:

CODE

Default choice: Round Robin (or Weighted Round Robin if servers differ)
- Works for 90% of use cases
- Simple to configure and understand

Choose Least Connections when:
- Request processing times vary significantly
- Some requests are fast (API calls) and some are slow (file uploads)
- Servers handle long-lived connections (WebSocket, streaming)

Choose IP Hash when:
- You need session affinity without external session store
- Servers maintain local caches that benefit from consistent routing

Choose Least Response Time when:
- Server performance varies due to different workloads
- You have monitoring infrastructure to measure response times

Interview Tip:
In most system design interviews, saying “round robin load balancing” is sufficient. If the interviewer presses for details, mention that you’d switch to least connections if request durations vary significantly, or weighted round robin if servers have different capacities.

SLIDE 82: Layer 4 vs Layer 7 Load Balancing

Layer 4 and Layer 7 Load Balancing

Load balancers operate at different layers of the network stack. The layer determines what information the load balancer can see and use for routing decisions.

Layer 4 (Transport Layer / TCP):

CODE

What it sees:
- Source IP address and port
- Destination IP address and port
- TCP/UDP protocol

What it does NOT see:
- HTTP method (GET, POST)
- URL path (/api/users vs /api/feed)
- HTTP headers (cookies, authorization)
- Request body

Routing decision based on:
- IP address
- Port number
- Simple algorithms (round robin, least connections)

How it forwards:
- Passes raw TCP packets to backend server
- Does NOT inspect or modify the content
- Very fast (no need to parse HTTP)

Layer 7 (Application Layer / HTTP):

Structure & Concept Flow

What it sees:

- Everything Layer 4 sees, plus:

- HTTP method (GET, POST, PUT, DELETE)

- URL path (/api/users/123)

- HTTP headers (Host, Authorization, Cookie, Content-Type)

- Query parameters (?page=2&sort=newest)

- Request body (JSON payload)

Routing decision based on:

- URL path: /api/users/* → User Service, /api/feed/* → Feed Service

- HTTP headers: route based on API version, A/B testing group

- Cookie/session: sticky sessions based on session ID

- Content type: different handling for images vs JSON

How it forwards:

- Terminates the HTTP connection

- Parses the request

- Makes a new connection to the backend

- Can modify headers, rewrite URLs, add headers

SLIDE 82B: Layer 4 vs Layer 7: Trade-Offs

Comparison:

Dimension	Layer 4 (TCP)	Layer 7 (HTTP)
Speed	Faster (no parsing)	Slower (must parse HTTP)
Throughput	Higher (millions of connections/sec)	Lower (hundreds of thousands/sec)
Routing intelligence	Basic (IP, port)	Advanced (URL, headers, cookies)
TLS termination	Pass-through or terminate	Terminate and inspect
Content-based routing	No	Yes (/api/users → User Service)
URL rewriting	No	Yes
WebSocket support	Yes (pass-through)	Yes (with upgrade handling)
Cost	Lower	Higher

When to Use Each:

CODE

Layer 4:
- High-throughput TCP services (database proxies, game servers)
- When you don't need content-based routing
- Internal load balancing between services (raw speed matters)
- When backend servers handle TLS themselves
- AWS NLB (Network Load Balancer) operates at Layer 4

Layer 7:
- HTTP/HTTPS web applications (most common case)
- Microservices routing (path-based routing to different services)
- A/B testing (route 10% of traffic to new version based on header)
- API gateway functionality (auth, rate limiting, URL rewriting)
- AWS ALB (Application Load Balancer) operates at Layer 7
- Nginx operates at Layer 7

In Practice:

Most production web applications use Layer 7 load balancers because they need content-based routing. The performance difference between Layer 4 and Layer 7 is irrelevant for most workloads. At extreme scale (millions of connections per second), Layer 4 is used for the first hop, with Layer 7 behind it for intelligent routing.

SLIDE 83: Health Checks and Failure Detection

Health Checks

A load balancer must know which backend servers are healthy (able to serve requests) and which are unhealthy (down, overloaded, or malfunctioning). Health checks are periodic probes sent to each server to verify its status.

How Health Checks Work:

Structure & Concept Flow

Load balancer configuration:

Health check endpoint: GET /health

Interval: 10 seconds

Timeout: 5 seconds

Healthy threshold: 3 consecutive successes

Unhealthy threshold: 2 consecutive failures

Every 10 seconds, load balancer sends:

GET /health → Server A → 200 OK (healthy ✓)

GET /health → Server B → 200 OK (healthy ✓)

GET /health → Server C → timeout (no response)

GET /health → Server C → timeout (2nd failure)

→ Server C marked unhealthy, removed from rotation

Types of Health Checks:

Type	What It Checks	Example
TCP	Can I open a connection?	Connect to port 8080
HTTP	Does the endpoint return 200?	GET /health returns 200 OK
Deep health	Is the application fully functional?	/health checks database, cache, disk
Custom	Application-specific logic	Check queue depth, memory usage, error rate

Shallow vs Deep Health Checks:

Structure & Concept Flow

Shallow health check (/health):

return 200 OK

Only confirms: "the process is running"

Does NOT confirm: "the application can serve real requests"

Deep health check (/health/deep):

1. Check database connection → OK

2. Check Redis connection → OK

3. Check disk space > 10% free → OK

4. Check memory usage < 90% → OK

5. Return 200 OK (all checks passed)

If any check fails → return 503 Service Unavailable

Trade-off:

Shallow: fast, low overhead, but misses real problems

Deep: catches real issues, but more expensive to run

Common approach: shallow checks every 5s, deep checks every 30s

Server Recovery:

Structure & Concept Flow

Server C was marked unhealthy.

After restart/fix:

GET /health → Server C → 200 OK (1st success)

GET /health → Server C → 200 OK (2nd success)

GET /health → Server C → 200 OK (3rd success, threshold met)

→ Server C marked healthy, added back to rotation

Why require multiple successes (healthy threshold = 3):

Prevents flapping. A server that responds once then fails again

should not receive real traffic. Requiring 3 consecutive successes

ensures the server is stably healthy.

SLIDE 84: Load Balancer Redundancy

Load Balancer Redundancy

The load balancer itself is a single point of failure. If it goes down, all traffic stops, even if every backend server is healthy. Production systems solve this with redundant load balancers.

Active-Passive Failover:

CODE

How it works:
1. Both load balancers share a virtual IP (VIP): 203.0.113.1
2. Active LB handles all traffic
3. Passive LB sends heartbeats to Active LB every 1-2 seconds
4. If Active LB stops responding (heartbeat timeout):
   - Passive LB takes over the virtual IP
   - Traffic automatically routes to the new active LB
   - Failover time: 5-30 seconds

The client sees the same IP address throughout.
The failover is invisible to the client.

Active-Active:

CODE

Both load balancers handle traffic simultaneously.
DNS returns both IPs, or a hardware device distributes between them.

Advantages over active-passive:
- Both LBs are utilized (no idle standby)
- Higher total throughput (2x a single LB)
- Faster failover (other LB is already warm)

Used by: AWS ALB (managed, multi-AZ), Cloudflare, large-scale deployments

Managed Load Balancers:

CODE

Cloud providers handle redundancy automatically:

AWS ALB/NLB:
- Deployed across multiple Availability Zones
- Automatic failover if one AZ goes down
- Auto-scales to handle traffic spikes
- No single point of failure
- Cost: ~$20/month + $0.008/GB processed

For most systems, using a managed load balancer is the right choice.
Building your own is only necessary for extreme customization needs.

SLIDE 85: Global Server Load Balancing (GSLB)

Global Server Load Balancing

GSLB distributes traffic across data centers in different geographic regions. While a regional load balancer distributes within a data center, GSLB operates at the global level using DNS.

How GSLB Works:

Structure & Concept Flow

User in Mumbai types youtube.com

DNS resolution with GSLB:

1. DNS query reaches YouTube's authoritative DNS

2. DNS server checks the user's location (via IP geolocation)

3. DNS returns the IP of the nearest data center:

- Mumbai user → Mumbai data center (34.93.100.50)

- London user → London data center (35.246.10.20)

- Virginia user → Virginia data center (52.23.45.67)

Result: User connects to the closest data center.

Latency: 5-20ms instead of 100-300ms (cross-continent).

Routing Policies:

Policy	How It Routes	Best For
Geographic	Nearest data center by location	Latency-sensitive applications
Latency-based	Lowest measured latency	When geographic proximity doesn’t equal lowest latency
Weighted	Percentage split across regions	Gradual migration to new region, A/B testing
Failover	Primary region, secondary on failure	Disaster recovery

GSLB Failover:

Structure & Concept Flow

Normal operation:

Mumbai users → Mumbai data center

London users → London data center

Mumbai data center goes down:

GSLB detects failure (health checks fail)

DNS updated: Mumbai users → Singapore data center (next closest)

TTL determines how quickly clients pick up the change:

TTL 60s: Most users re-route within 1-2 minutes

TTL 300s: Most users re-route within 5-10 minutes

This is why DNS TTL matters for failover.

Lower TTL = faster failover but more DNS queries.

Technologies:

Service	Provider
Route 53	AWS (supports geolocation, latency, weighted, failover)
Azure Traffic Manager	Azure
Cloud DNS	Google Cloud
Cloudflare DNS	Cloudflare

SLIDE 86: Load Balancing: Common Interview Mistakes

Common Interview Mistakes

Structure & Concept Flow

Mistake 1: Forgetting the load balancer entirely

Drawing: Client → Server (single server, no LB)

Problem: Single point of failure, no horizontal scaling.

Better: Always include a load balancer between clients and

application servers, even if you only draw one server behind it.

It signals that you're designing for scalability.

Mistake 2: Not explaining what happens when a server fails

"We have 3 servers behind a load balancer."

Interviewer: "What happens if Server 2 goes down?"

Bad answer: "The system degrades."

Better: "The load balancer detects the failure via health checks

within 10-30 seconds, removes Server 2 from the rotation, and

distributes its traffic to Servers 1 and 3. Users experience

no downtime. Once Server 2 recovers and passes 3 consecutive

health checks, it's added back."

Mistake 3: Using sticky sessions as the default

"We'll use IP hash so each user goes to the same server."

Problem: This implies your servers are stateful.

Better: "Our servers are stateless. All session data is in Redis.

Any server can handle any request. We use round robin for

even distribution."

Mistake 4: Not addressing load balancer redundancy

Interviewer: "What if the load balancer itself goes down?"

Bad answer: "That shouldn't happen."

Better: "We use active-passive failover with a shared virtual IP.

If the active LB fails, the passive takes over within 10-30 seconds.

In practice, we'd use a managed service like AWS ALB which handles

redundancy automatically across availability zones."

SLIDE 87: Section Summary and Interview Tips

Section 7 Summary: Load Balancing

Concepts Covered:

Concept	One-Line Summary
Load balancer role	Distributes traffic, detects failures, terminates TLS
Placement	Edge (global), regional (HTTP), internal (service-to-service)
Round robin	Equal distribution, simplest algorithm
Weighted round robin	Proportional to server capacity
Least connections	Routes to least busy server, best for variable request times
IP hash	Same client always goes to same server
Layer 4	Fast TCP-level routing, no content inspection
Layer 7	HTTP-aware routing by URL, headers, cookies
Health checks	Periodic probes to detect unhealthy servers (shallow and deep)
Redundancy	Active-passive or active-active to eliminate LB as single point of failure
GSLB	DNS-based routing to nearest data center globally

Key Numbers to Remember:

CODE

- Health check interval: 5-30 seconds
- Unhealthy threshold: 2-3 consecutive failures
- Healthy threshold: 3 consecutive successes
- Active-passive failover: 5-30 seconds
- Managed LB cost: ~$20/month + per-GB charges
- Layer 4 throughput: millions of connections/sec
- Layer 7 throughput: hundreds of thousands of requests/sec

How This Connects to Other Sections:

CODE

Section 3 (Servers):   Stateless servers are what make load balancing work
Section 5 (Scaling):   Load balancer is Stage 3 of the scaling progression
Section 8 (CDN):       CDN is essentially a global load balancer for static content
Section 11 (Distributed): Consistent hashing is used in some LB algorithms
Section 13 (Design):   URL shortener includes LB in the architecture diagram