TRADE-OFFS & COST ANALYSIS

SLIDE 53: Key Trade-Offs (Part 1)

Step 8: Trade-Offs and Design Decisions

Trade-Off 1: Consistency vs Availability (CAP Theorem)

Implications:

• User might not see their own post immediately in feed (replication lag)
• This is acceptable - better than system being down
• Critical operations (money, security) would need stronger consistency

SLIDE 53B: Key Trade-Offs (Part 2)

Trade-Off 2: SQL vs NoSQL

Why SQL for Instagram:

• User-post-follow relationships are inherently relational
• Need for transactions (likes, follows)
• Complex queries for feed generation
• Team expertise and tooling maturity

When NoSQL Would Be Better:

• Unstructured data (logs, events)
• Extreme write throughput (millions of writes/sec)
• Simple key-value lookups only

SLIDE 53C: Key Trade-Offs (Part 3)

Trade-Off 3: Fan-Out on Write vs Fan-Out on Read

Why Hybrid:

• 99% of users benefit from fast reads (fan-out on write)
• 1% of celebrities don’t cause system overload
• Slight latency increase for users following celebrities (acceptable)

SLIDE 53D: Key Trade-Offs (Part 4)

Trade-Off 4: Synchronous vs Asynchronous Processing

Philosophy: Async by default unless immediate feedback critical for UX

SLIDE 53E: Key Trade-Offs (Part 5)

Trade-Off 5: Denormalization for Performance

Normalized (Slower):

-- Count followers every time
SELECT COUNT(*) FROM follows WHERE followee_id = 12345
-- Slow for users with millions of followers

Denormalized (Faster):

-- Store count in users table
SELECT follower_count FROM users WHERE user_id = 12345
-- Instant, but must maintain count (eventual consistency)

Decision: Denormalize counts (followers, likes, comments)

• Trade: Storage space and consistency complexity
• Benefit: 1000x faster queries

SLIDE 53F: Key Trade-Offs (Part 6)

Trade-Off 6: Monolith vs Microservices

Why Microservices for Instagram:

• Feed Service needs 100x more resources than User Service
• Different teams can work independently
• Fault isolation (Media Service down ≠ Feed Service down)

When Monolith Makes Sense:

• Startup with <100K users
• Single team
• Unclear product-market fit

SLIDE 54: Alternative Architectures (Part 1)

Alternative Approaches Not Chosen

Alternative 1: NoSQL Database (Cassandra / DynamoDB)

How It Would Work:

Posts table in Cassandra:
Partition key: user_id
Clustering key: created_at (descending)

Benefits:
- Native horizontal scaling (add nodes easily)
- High write throughput
- No need for manual sharding

Drawbacks:
- Limited query patterns (must query by partition key)
- No joins (must denormalize everything)
- Complex for relational data (user-follow graph)
- Eventual consistency harder to reason about

SLIDE 54B: Alternative Architectures (Part 2)

Query Examples:

Query:
SELECT * FROM posts WHERE user_id = 12345
✓ Efficient (single partition)

SELECT * FROM posts WHERE caption CONTAINS 'sunset'
✗ Impossible (full table scan, not supported)

Why We Didn’t Choose This:

• Instagram’s data is inherently relational (users, posts, follows, likes)
• Complex queries needed for feed generation
• ACID transactions needed for critical operations
• SQL better fits the problem domain

When NoSQL Would Be Better:

• Simple key-value lookups only
• No complex relationships
• Extreme write throughput requirements

SLIDE 54C: Alternative Architectures (Part 3)

Alternative 2: GraphQL API (Instead of REST)

How It Would Work:

query GetFeed {
  feed(limit: 20) {
    posts {
      id
      image_url
      caption
      user {
        id
        username
        profile_pic_url
      }
      likes_count
      comments_count
      liked_by_me
    }
  }
}

Benefits:
- Client specifies exact fields needed (no over-fetching)
- Single endpoint (not /feed, /posts, /users separately)
- Better for mobile (reduces data transfer)
- Strongly typed schema

Drawbacks:
- More complex to implement
- Harder to cache (dynamic queries)
- Learning curve for team
- Over-fetching can become under-fetching problem (N+1 queries)

SLIDE 54D: Alternative Architectures (Part 4)

Why We Didn’t Choose This (for MVP):

• REST is simpler for initial implementation
• Caching is easier with REST (fixed endpoints)
• Team familiarity with REST
• Can add GraphQL layer later (Facebook/Instagram did this)

When GraphQL Makes Sense:

• Mobile-first applications (bandwidth optimization)
• Many different client types (web, mobile, third-party)
• Mature product with stable API

SLIDE 54E: Alternative Architectures (Part 5)

Alternative 3: Serverless Architecture (AWS Lambda / Azure Functions)

How It Would Work:

SLIDE 54F: Alternative Architectures (Part 6)

Why We Didn’t Choose This:

• Cold start latency unacceptable for user-facing APIs
• Complex for stateful operations (WebSocket connections)
• Less control over performance optimization
• At Instagram’s scale, dedicated servers more cost-effective

When Serverless Makes Sense:

• Spiky workloads (not constant high traffic)
• Background jobs (image processing could use Lambda)
• Startups wanting fast iteration

SLIDE 54G: Alternative Architectures (Part 7)

Alternative 4: Event-Driven Architecture (Event Sourcing)

How It Would Work:

SLIDE 54H: Alternative Architectures (Part 8)

Why We Didn’t Choose This (for MVP):

• Adds significant complexity
• Eventual consistency harder to reason about
• Overkill for initial launch

When Event-Driven Makes Sense:

• Mature product with many services
• Need for audit log / event replay
• Analytics-heavy workloads

SLIDE 55: Failure Scenarios (Part 1)

Handling Failures: Production Reality

Failure Scenario 1: Database Shard Goes Down

Impact:

• Users on that shard (2M users) cannot read/write data
• ~0.1% of total users affected

Detection:

• Health checks fail (10-30 seconds to detect)
• Monitoring alerts fire
• On-call engineer paged

Automatic Mitigation:

1. Primary database fails
2. Monitoring detects failure (30 seconds)
3. Automatic failover to read replica
4. Promote replica to new primary (30-60 seconds)
5. Update application config (10 seconds)
6. Traffic flows to new primary
Total downtime: 1-2 minutes

Recovery:

1. Investigate root cause (disk failure, network issue, etc.)
2. Provision new replica to replace failed primary
3. Catch up replication
4. Return to normal configuration
Total recovery time: 30-60 minutes

SLIDE 55B: Failure Scenarios (Part 2)

Failure Scenario 2: CDN Region Failure

Impact:

• Users in that region see slow image loads
• Degraded experience, not complete outage

Automatic Mitigation:

SLIDE 55C: Failure Scenarios (Part 3)

Failure Scenario 3: Feed Service Crashes

Impact:

• Users cannot load feeds
• Other functionality (posting, liking) still works

Automatic Mitigation:

1. Feed Service instance crashes
2. Health check fails (10 seconds)
3. Load balancer removes instance from pool
4. Traffic routes to healthy instances
5. Auto-scaler detects high load
6. Spins up new instances (2-3 minutes)
Total downtime: 10-30 seconds for affected requests

SLIDE 55D: Failure Scenarios (Part 4)

Failure Scenario 4: Redis Cache Cluster Failure

Impact:

• Feed generation becomes slow (no cache)
• Increased database load

Degraded Mode Operation:

Recovery:

1. Restore Redis cluster (10-30 minutes)
2. Cache slowly rebuilds (cache misses)
3. Gradually return to normal performance
Total degraded time: 30-60 minutes

SLIDE 55E: Failure Scenarios (Part 5)

Failure Scenario 5: Total Data Center Outage

Impact:

• Catastrophic if single-region deployment
• All services in that region down

Multi-Region Mitigation:

Instagram deploys to multiple regions:
- Primary: us-east-1 (Virginia)
- Secondary: us-west-2 (Oregon)
- Tertiary: eu-west-1 (Ireland)

Disaster recovery:
1. us-east-1 completely fails (earthquake, power outage)
2. Automated failover to us-west-2 (10-30 minutes)
3. DNS updated to point to us-west-2
4. Some data loss acceptable (replication lag: 1-5 minutes)
5. Service restored

Total downtime: 10-30 minutes
Data loss: Last 1-5 minutes of writes

SLIDE 55F: Failure Scenarios (Part 6)

Monitoring and Alerting:

Metrics monitored (every 10 seconds):
- API latency (P50, P95, P99)
- Error rate (5xx responses)
- Database query time
- Cache hit rate
- Queue length
- Server CPU/memory usage

Alerts triggered when:
- P99 latency > 1 second for 5 minutes
- Error rate > 1% for 2 minutes
- Database unavailable
- Cache hit rate < 50%

Alert channels:
- PagerDuty (on-call engineer)
- Slack (team notification)
- Email (management)

SLIDE 55G: Failure Scenarios (Part 7)

Graceful Degradation:

When system under stress:
1. Disable non-critical features first
   - Explore page (keep feed working)
   - Recommendations (keep basic feed)
   - Stories (keep main posts)

2. Reduce quality
   - Serve lower-res images
   - Reduce feed size (10 posts instead of 20)
   - Disable real-time updates

3. Last resort: Read-only mode
   - Disable posting
   - Allow viewing only
   - Protect database from writes

SLIDE 56: Cost Optimization (Part 1)

Cost Considerations at Instagram Scale

Monthly Cost Estimate (Rough Order of Magnitude):

Annual Cost: ~$680 million

SLIDE 56B: Cost Optimization (Part 2)

Cost Optimization Strategies:

1. Storage Tier Optimization (Saves $30M+/year)

Without lifecycle policies:
All photos in S3 Standard: 100 PB × $23/TB × 12 months = $27.6M/year

With lifecycle policies:
Year 1 photos (hot): 20 PB × $23 = $460K/month
Year 2 photos (warm): 20 PB × $12.50 = $250K/month
Year 3+ photos (cold): 60 PB × $4 = $240K/month
Total: $950K/month = $11.4M/year

Savings: $16.2M/year (60% reduction)

SLIDE 56C: Cost Optimization (Part 3)

2. CDN Optimization (Saves $10M+/year)

Increase cache hit ratio from 80% to 85%:

Origin egress reduction: 5% of 600 PB = 30 PB/month
Cost saving: 30,000 TB × $90/TB = $2.7M/month = $32.4M/year

How to achieve:
- Longer cache TTL (1 year instead of 1 day)
- Pre-warming cache for popular content
- Image compression (smaller files = more cacheable)

SLIDE 56D: Cost Optimization (Part 4)

3. Reserved Instances (Saves $5M+/year)

Pay upfront for 1-year commitment:

On-demand pricing: 1,000 servers × $200 = $200K/month
Reserved (1-year): 1,000 servers × $120 = $120K/month
Savings: $80K/month × 12 = $960K/year per 1,000 servers

For entire fleet (5,000 instances):
Savings: $4.8M/year

4. Image Compression (Saves $3M+/year)

Current: 2MB average per photo
Improved compression: 1.5MB per photo (25% reduction)

Storage savings:
100M photos/day × 365 days × 0.5MB = 18.25 PB/year saved
Cost: 18,250 TB × $23/TB = $420K/year storage
Plus CDN bandwidth savings: ~$2M/year

Total: ~$3M/year

Implementation: Use WebP format instead of JPEG

SLIDE 56E: Cost Optimization (Part 5)

5. Database Right-Sizing (Saves $8M+/year)

Over-provisioned:
4,000 instances × $500 = $2M/month

Right-sized (based on actual usage):
- Reduce instances by 30% in off-peak hours
- Use smaller instance types for low-traffic shards
- Savings: $600K/month = $7.2M/year

6. Spot Instances for Workers (Saves $2M+/year)

Image processing workers (non-critical):
- Can tolerate interruptions
- Use spot instances (70% discount)

On-demand: 5,000 workers × $100 = $500K/month
Spot instances: 5,000 workers × $30 = $150K/month
Savings: $350K/month = $4.2M/year

SLIDE 56F: Cost Optimization (Part 6)

Total Potential Savings: $40-50M annually (7-9% cost reduction)

ROI on Engineering Effort:

• 2 engineers for 3 months on optimization: $100K cost
• Annual savings: $40M
• ROI: 400:1

Cost Per User Metrics:

Total monthly cost: $56.57M
Monthly active users: 500M
Cost per user: $0.11/month

Daily active users: 500M
Cost per DAU: $0.11/month

For comparison:
- Twitter: ~$0.20/user/month
- Facebook: ~$0.15/user/month
- Instagram: ~$0.11/user/month (estimated)

Lower is better (more efficient)