SLIDE 115: Section: Multi-Region and Global Scale
Serving Users Worldwide
Structure & Concept Flow
graph LR
A[US-East] --> B[US-West]
A --> C[EU-West]
A --> D[AP-Southeast]
B --> C
C --> D
Structure & Concept Flow
**Why Multi-Region:**
| **Requirement** | **Solution** |
| --- | --- |
| Low latency worldwide | Deploy close to users |
| High availability | Survive region failures |
| Data sovereignty | Keep data in required regions |
| Disaster recovery | Geographic redundancy |
**Key Insight:**
Multi-region is not just “deploy in more places.” It fundamentally changes how you think about data consistency, failover, and operational complexity.
---
## SLIDE 116: Multi-Region: Latency Reality
**Physics Matters**
**Speed of Light Constraints:**
| **Route** | **Distance** | **Min Latency** |
| --- | --- | --- |
| NYC to London | 5,500 km | 37ms |
| NYC to Tokyo | 10,800 km | 72ms |
| London to Sydney | 17,000 km | 113ms |
These are minimums. Real latency is 2-3x higher.
**Impact on User Experience:**
```
Single region (US-East):
- US users: 20ms latency
- EU users: 100ms latency
- Asia users: 200ms latency
Multi-region:
- US users: 20ms (US-East)
- EU users: 20ms (EU-West)
- Asia users: 20ms (AP-Southeast)
```
**Rule of Thumb:**
Every 100ms of latency reduces conversions by 1%.
---
## SLIDE 117: Multi-Region: Architecture Patterns
**Choosing Your Approach**
**Pattern 1: Active-Passive**
```
graph LR
A[Users] --> B[Primary: US-East]
B --> C[Database Primary]
C -->|Replication| D[Database Replica]
D --> E[Secondary: EU-West Standby]
```
Primary handles all traffic. Secondary is for disaster recovery only.
**Pattern 2: Active-Active**
```
graph LR
A[US Users] --> B[US-East]
C[EU Users] --> D[EU-West]
B --> E[US Database]
D --> F[EU Database]
E <-->|Sync| F
```
Both regions handle traffic. Data synchronized between them.
---
## SLIDE 118: Multi-Region: Architecture Patterns (continued)
**Pattern Comparison:**
| **Aspect** | **Active-Passive** | **Active-Active** |
| --- | --- | --- |
| Complexity | Lower | Higher |
| Cost | Lower | Higher |
| Latency | Higher for distant users | Low for all users |
| Failover time | Minutes | Seconds |
| Data consistency | Simpler | Complex |
| Use case | DR-focused | Global user base |
**Pattern 3: Follow-the-Sun**
```
06:00-18:00 UTC: EU-West primary
18:00-06:00 UTC: US-East primary
Traffic naturally follows daylight hours.
Maintenance during region's night.
```
---
## SLIDE 119: Multi-Region: Data Replication Strategies
**Keeping Data in Sync**
**Synchronous Replication:**
```
graph LR
A[Write] --> B[Primary]
B --> C[Wait for Replica ACK]
C --> D[Replica]
D --> E[ACK]
E --> F[Commit]
```
**Characteristics:**
| **Aspect** | **Value** |
| --- | --- |
| Consistency | Strong |
| Write latency | High (round trip to replica) |
| Data loss on failure | None |
| Use case | Financial transactions |
**Asynchronous Replication:**
```
graph LR
A[Write] --> B[Primary]
B --> C[Commit]
B --> D[Async Replicate]
D --> E[Replica]
```
| **Aspect** | **Value** |
| --- | --- |
| Consistency | Eventual |
| Write latency | Low |
| Data loss on failure | Possible (lag window) |
| Use case | Most applications |
---
## SLIDE 120: Multi-Region: Data Replication Strategies (continued)
**Conflict Resolution:**
**Problem: Concurrent Writes to Same Record**
```
US-East: UPDATE user SET name = 'Alice' WHERE id = 1
EU-West: UPDATE user SET name = 'Bob' WHERE id = 1
Both happen at same time. Which wins?
```
**Resolution Strategies:**
| **Strategy** | **How It Works** | **Trade-off** |
| --- | --- | --- |
| Last Write Wins | Timestamp comparison | May lose updates |
| First Write Wins | Reject later writes | Requires retry |
| Merge | Combine changes | Complex logic |
| Custom | Application-specific | Development effort |
**Last Write Wins Implementation:**
```
def merge_record(local, remote):
if remote.updated_at > local.updated_at:
return remote
else:
return local
```
---
## SLIDE 121: Multi-Region: Data Replication Strategies (continued)
**CRDTs: Conflict-Free Replicated Data Types**
```
# Counter CRDT: Each region tracks its own increments
class GCounter:
def __init__(self, region_id):
self.region_id = region_id
self.counts = {} # region -> count
def increment(self):
self.counts[self.region_id] = self.counts.get(self.region_id, 0) + 1
def value(self):
return sum(self.counts.values())
def merge(self, other):
# Take max for each region
for region, count in other.counts.items():
self.counts[region] = max(self.counts.get(region, 0), count)
```
**CRDT Use Cases:**
| **Data Type** | **CRDT** | **Example** |
| --- | --- | --- |
| Counters | G-Counter | Like counts |
| Sets | G-Set | Tags, members |
| Registers | LWW-Register | User preferences |
| Maps | OR-Map | Shopping cart |
---
## SLIDE 122: Multi-Region: Global Load Balancing
**Routing Users to Nearest Region**
**DNS-Based Routing:**
```
graph LR
A[User DNS Query] --> B[Global DNS]
B -->|US User| C[us-east.example.com]
B -->|EU User| D[eu-west.example.com]
B -->|Asia User| E[ap-southeast.example.com]
```
**Routing Policies:**
| **Policy** | **How It Works** | **Use Case** |
| --- | --- | --- |
| Geolocation | Route by user location | Serve nearest region |
| Latency | Route to lowest latency | Performance optimization |
| Weighted | Distribute by percentage | Gradual migration |
| Failover | Primary/secondary | Disaster recovery |
---
## SLIDE 123: Multi-Region: Global Load Balancing (continued)
**Anycast Routing:**
```
Same IP address announced from multiple locations.
Network routes to nearest announcement.
User in London -> London POP
User in Tokyo -> Tokyo POP
Same IP: 203.0.113.1
```
**Implementation Example (AWS Route 53):**
```
{
"Name": "api.example.com",
"Type": "A",
"SetIdentifier": "us-east",
"Region": "us-east-1",
"TTL": 60,
"ResourceRecords": [{ "Value": "10.0.1.1" }],
"HealthCheckId": "abc123"
}
```
**Health-Based Routing:**
```
def get_healthy_endpoints():
endpoints = []
for region in REGIONS:
health = check_region_health(region)
if health.status == "healthy":
endpoints.append({
"region": region,
"weight": health.capacity_percent
})
return endpoints
```
---
## SLIDE 124: Multi-Region: Regional Failover
**Handling Region Outages**
**Failover Process:**
```
graph LR
A[Region Failure Detected] --> B[Health Checks Fail]
B --> C[Update DNS/LB]
C --> D[Traffic to Surviving Region]
D --> E[Monitor and Recover]
```
**Failover Timeline:**
| **Phase** | **Duration** | **Action** |
| --- | --- | --- |
| Detection | 30-60 seconds | Health checks fail |
| Decision | 1-5 minutes | Confirm not transient |
| DNS Update | 1-5 minutes | Propagation time |
| Traffic Shift | 5-15 minutes | Clients get new IPs |
| Full Failover | 10-30 minutes | All traffic migrated |
---
## SLIDE 125: Multi-Region: Regional Failover (continued)
**Automated Failover:**
```
class RegionFailover:
def __init__(self, regions, health_threshold=0.5):
self.regions = regions
self.health_threshold = health_threshold
def check_and_failover(self):
for region in self.regions:
health = self.check_health(region)
if health.success_rate < self.health_threshold:
if self.confirm_failure(region):
self.execute_failover(region)
def confirm_failure(self, region):
# Multiple checks from different locations
checks = [
self.check_from_location(region, "us-east"),
self.check_from_location(region, "eu-west"),
self.check_from_location(region, "ap-south"),
]
# Majority must agree
return sum(not c.healthy for c in checks) >= 2
def execute_failover(self, failed_region):
# Update DNS to remove failed region
dns.remove_region(failed_region)
# Scale up surviving regions
for region in self.regions:
if region != failed_region:
scale_up(region, factor=1.5)
# Alert operations
alert_oncall(f"Failover executed:{failed_region}")
```
---
## SLIDE 126: Multi-Region: Data Sovereignty
**Keeping Data Where It Belongs**
**Regulatory Requirements:**
| **Regulation** | **Requirement** |
| --- | --- |
| GDPR (EU) | EU user data must be processable in EU |
| Data Localization (Russia) | Data must stay in Russia |
| Data Localization (China) | Data must stay in China |
| HIPAA (US) | Healthcare data protection |
**Architecture for Data Sovereignty:**
```
graph LR
A[EU Users] --> B[EU Region]
B --> C[EU Database]
D[US Users] --> E[US Region]
E --> F[US Database]
G[Global Services] --> H[Metadata Only]
```
**Implementation:**
```
def get_database_for_user(user):
if user.country in EU_COUNTRIES:
return eu_database
elif user.country in APAC_COUNTRIES:
return apac_database
else:
return us_database
def store_user_data(user, data):
db = get_database_for_user(user)
db.store(user.id, data)
```
---
## SLIDE 127: Multi-Region: Data Sovereignty (continued)
**Cross-Border Data Handling:**
```
class SovereignDataManager:
def get_user_profile(self, user_id, requesting_region):
user = self.get_user_region(user_id)
if user.data_region == requesting_region:
# Same region, return full data
return self.db[user.data_region].get_full_profile(user_id)
else:
# Cross-region, return only non-PII
return self.db[user.data_region].get_public_profile(user_id)
def process_request(self, user_id, operation, requesting_region):
user = self.get_user_region(user_id)
if self.requires_local_processing(operation):
# Must process in user's region
return self.remote_call(
region=user.data_region,
operation=operation,
user_id=user_id
)
else:
# Can process locally
return self.local_process(operation, user_id)
```
---
## SLIDE 128: Multi-Region: Testing
**Validating Multi-Region Setup**
**What to Test:**
| **Test Type** | **What It Validates** |
| --- | --- |
| Latency tests | Users get routed to nearest region |
| Failover drills | Traffic shifts when region fails |
| Replication tests | Data syncs correctly |
| Conflict tests | Concurrent writes handled |
| Sovereignty tests | Data stays in correct region |
**Chaos Engineering for Multi-Region:**
```
def region_failure_test():
# Baseline metrics
baseline = collect_metrics(duration=10_minutes)
# Simulate region failure
block_region_traffic("us-east")
# Measure impact
during_failure = collect_metrics(duration=30_minutes)
# Restore
unblock_region_traffic("us-east")
# Validate
assert during_failure.error_rate < 0.01 # <1% errors
assert during_failure.latency_p99 < baseline.latency_p99 * 2
assert during_failure.requests_per_second > baseline.rps * 0.9
```
---
## SLIDE 129: Multi-Region: Testing (continued)
**Game Day Exercises:**
```
MULTI-REGION GAME DAY RUNBOOK
Scenario: US-East region complete failure
1. PREPARATION (Day before)
- Notify stakeholders
- Ensure oncall coverage
- Verify rollback capability
2. EXECUTION
09:00 - Begin monitoring baseline
09:30 - Announce test start
09:35 - Simulate US-East failure
09:40 - Observe automatic failover
09:50 - Verify EU-West handling traffic
10:00 - Check data consistency
10:15 - Restore US-East
10:30 - Verify traffic rebalances
10:45 - End test
3. POST-MORTEM
- What worked?
- What failed?
- What can we automate?
- Update runbooks
```
---
## SLIDE 130: Multi-Region: Cost Considerations
**The Price of Global Scale**
**Cost Multipliers:**
| **Component** | **Single Region** | **Multi-Region (3)** |
| --- | --- | --- |
| Compute | 1x | 2-3x |
| Database | 1x | 2-3x |
| Storage | 1x | 2-3x |
| Data transfer | Low | High (replication) |
| Operations | 1x | 1.5-2x |
**Cost Optimization Strategies:**
```
# Scale each region based on local traffic patterns
REGION_SCHEDULES = {
"us-east": {
"peak_hours": (9, 21), # 9am-9pm EST
"peak_replicas": 10,
"off_peak_replicas": 3
},
"eu-west": {
"peak_hours": (8, 20), # 8am-8pm CET
"peak_replicas": 8,
"off_peak_replicas": 2
},
"ap-southeast": {
"peak_hours": (9, 21), # 9am-9pm SGT
"peak_replicas": 5,
"off_peak_replicas": 2
}
}
```
---
## SLIDE 131: Multi-Region: Summary
**Key Takeaways**
**Architecture Decisions:**
- Active-passive for DR focus
- Active-active for global latency requirements
- Consider data sovereignty requirements early
**Data Replication:**
- Synchronous for strong consistency (higher latency)
- Asynchronous for performance (eventual consistency)
- Plan conflict resolution strategy
**Operations:**
- Automate failover with safeguards
- Test failover regularly (monthly)
- Monitor cross-region replication lag
**Costs:**
- Multi-region is 2-3x more expensive
- Optimize with regional scaling
- Minimize cross-region data transfer
---