Backend Scaling and Performance Engineering: Part-1

This comprehensive video dives into the foundational and advanced concepts of performance and scaling as they relate to backend systems. It covers critical metrics like latency and throughput, explains common performance pitfalls, database optimizations, caching strategies, and compares vertical and horizontal scaling approaches. The content equips backend engineers with the language, intuition, and practical tools to analyze and optimize system behavior under load.

1. Core Performance Metrics: Latency and Throughput

Latency and throughput are fundamental to understanding system performance but represent different measurements:

• Latency: Total time elapsed from a user's action (e.g., button click) until the response is fully rendered.

  • Includes network requests, server processing, database access, API calls, and browser rendering.
  • Latency fluctuates per request, influenced by caching, server load, network conditions.
  • Average latency can be misleading because it masks outliers where some requests experience severe delays.
  • Instead, percentiles (P50, P90, P99) provide better insight:
    • P50: 50% of users experience latency below this value.
    • P90: 90% are quicker, 10% slower than this latency.
    • P99: 99% receive faster responses, 1% experience higher latency.

• Throughput: Number of requests a system can handle per unit time (e.g., requests/second).

  • Latency and throughput are inherently related but non-linear.
  • At low throughput, latency remains low; as throughput approaches system capacity, latency increases sharply.
  • Essential for predicting how a system handles traffic spikes like Black Friday or viral campaigns.

Key insight: Focus on latency percentiles, especially P95 and P99, since these represent requests involving complex operations impacting valuable users and business logic.

2. Utilization vs Latency: Queues and Capacity Headroom

The video uses analogies (ice cream shop, highway traffic) to illustrate system behavior as utilization increases:

• Utilization: Percentage of system capacity actively in use.
  • At low utilization, requests are handled immediately with low latency.
  • As utilization increases, requests queue up, increasing latency.
  • Latency grows exponentially near 100% utilization rather than linearly.

• Systems typically run at 60-80% utilization, reserving headroom to absorb bursty traffic.
• Traffic is bursty and non-uniform; peak bursts can exceed capacity leading to crashes without buffer.

Conclusion: Never operate at saturated capacity; always reserve headroom to maintain stability and acceptable latency.

3. Identifying Bottlenecks: Measure, Don’t Guess

Common pitfall: Engineers guess bottlenecks (database, caching) and apply standard fixes without proper measurement, wasting time and effort.

• Example: Adding caching to an update API that doesn’t benefit from caching since data changes frequently.
• Granular timing and logging can pinpoint true bottlenecks, e.g., synchronous logging to remote services adding hundreds of milliseconds delay.
• Rule: Always add instrumentation and profile before choosing optimizations.

Profiling Tools

• Profilers: Measure CPU-bound function execution times and call stacks (e.g., flame graphs).
  • Better for CPU-heavy areas (business logic, computations).
• Distributed Tracing: Tracks entire lifecycle of a request passing through components (DB queries, external API calls).
  • Especially useful for IO-bound activities, typical in backend apps.
  • Enables precise latency breakdown of external calls, database operations, and internal logic.

4. Database Performance: Avoiding the N+1 Query Problem and Effective Indexing

N+1 Query Problem

• Occurs when fetching lists with associated data results in one query for the list and $\mathbf{n}$ additional queries for related data.
• Linear number of queries to number of items drastically slows response times.
• Solution: Bulk fetching via joins or ORM prefetching techniques reduces queries to a constant number (e.g., 2 queries regardless of $\mathbf{n}$).

Modern ORMs (e.g., Django, Rails, Prisma) provide primitives to minimize this problem.

Indexing

• Indexes vastly improve query speeds by avoiding full table scans.
• Indexes store sorted mappings (usually B-trees) to accelerate lookups.
• However, indexes incur costs:
  • Additional storage space.
  • Slower writes (INSERT/UPDATE/DELETE) due to maintaining index structures.
• Choose indexes strategically, based on query patterns identified via monitoring and explain/analyze tools.
• Types of indexes:
  • Composite Index: Multi-column index that optimizes queries filtering on several columns. Order matters for matching queries.
  • Covering Index: Index contains all data needed for a query, avoiding fetching from table (reduces IO).

Best practice: Add necessary indexes upfront for commonly queried columns and add others based on query plans during production monitoring.

5. Database Connections: Connection Pooling

Establishing database connections is expensive due to TCP handshakes, authentication, and memory allocation.

Problems without pooling:

• High latency on each query due to connection setup.
• Risk of exhausting database connection limits under load (e.g., Postgres limits around 400-600 connections).

Connection Pooling

• Maintains a set of open, reusable connections.
• Server borrows a connection from the pool, executes queries, and returns it without closing.
• Reduces connection overhead and prevents connection exhaustion.

Two types:

• Internal Pooling: Each instance maintains its own pool.
  • Problematic with horizontal scaling; multiple servers create large cumulative connections.
• External Pooling: A shared pool broker like PG Bouncer mediates database access for all servers.
  • External poolers reduce connection overload and handle traffic spikes gracefully.

6. Caching: Strategies, Challenges, and Patterns

Importance of Caching

• Caching stores results of expensive operations (usually DB queries).
• Proper caching reduces latency significantly (e.g., from 800 ms to 50 ms).

Cache Invalidation Challenges

• Synchronizing cache with underlying data is notoriously difficult.
• Two main invalidation strategies:
  • Time-based invalidation: Cache expires after preset TTL.
    • Straightforward but choosing TTL optimally is challenging; too long leads to stale data.
  • Event-based invalidation: Cache is explicitly invalidated upon data changes.
    • More accurate but requires discipline to implement cache deletion on every write.

Cache Location

• Typical setups combine both via tiered caching:
  • Small local cache for “hot” data.
  • Upstream distributed cache for broader consistency.

Caching Patterns

Cache Hit Rate

• Percentage of requests served from cache.
• High hit rate (e.g., 90%) indicates effective caching.
• Influenced by TTL, cache size, and data access patterns.
• Poor understanding of access patterns leads to lower hit rates and poor caching efficiency.

7. Scaling Approaches: Vertical vs. Horizontal Scaling

Vertical Scaling (Scaling Up)

• Increase capacity of a single server (CPU cores, RAM, storage, network).
• Simple, intuitive, no changes to application architecture required.
• Doubling resources roughly doubles capacity.
• Economically often cheaper than multiple smaller machines.
• Downsides:
  • Hard upper limit due to hardware caps.
  • Single point of failure: server crash causes full downtime.
  • No geo-distribution: all traffic routed to one location, causing increased latency for distant users.

Horizontal Scaling (Scaling Out)

• Add more instances of servers running the same application.
• Load balanced to share incoming traffic.
• Theoretically unlimited capacity; scale out as needed.
• Provides:
  • Redundancy (fault tolerance).
  • Geographic distribution by deploying servers near users.
• Challenges:
  • Infrastructure complexity increases.
  • Requires load balancing components.
  • Need to ensure data consistency and synchronization between instances.
  • Handling server health, failures, network partitioning, and routing logic.
• Tradeoff: reduces single points of failure but introduces new distributed systems complexities, requiring careful design.

Summary

• Latency and throughput metrics are central to understanding and optimizing backend performance, with latency percentiles providing richer insights than averages.
• System utilization vs latency relationship is highly non-linear, demanding headroom for stable performance.
• Diagnose performance bottlenecks through measurement and profiling, not guesswork.
• Database optimizations due to N+1 query problems and proper indexing critically impact latency.
• Connection pooling reduces overhead and prevents hitting database connection limits at scale.
• Caching is vital but comes with difficult cache invalidation problems; correct pattern choice depends on use case.
• Vertical scaling offers simplicity but limited capacity and fault tolerance, while horizontal scaling provides scalability and redundancy at the cost of architecture complexity.

Understanding these fundamentals enables practical, reliable decision-making when designing backend systems that scale efficiently and perform consistently under load.