Backend Scaling and Performance Engineering: Part-2

This video provides an in-depth explanation of key concepts and architectural patterns for scaling backend applications effectively. It systematically covers statelessness in horizontal scaling, load balancers and their algorithms, database scaling approaches including read replicas and sharding, the role of CDNs with edge computing, asynchronous processing benefits, and microservices advantages and complexities. The discussion integrates underlying principles, common pitfalls, and practical solutions for building scalable, performant, and maintainable backend infrastructures.

1. Statelessness as the Foundation of Horizontal Scaling

Horizontal scaling involves adding multiple instances of an application backend rather than increasing the capacity of a single server (vertical scaling). The fundamental enabler of horizontal scaling is statelessness.

Statelessness means no server instance holds exclusive or session-specific data internally.
If any backend instance maintains data only accessible to itself (stateful behavior), horizontal scaling will cause errors due to inconsistent data access.
Statelessness guarantees that requests can be sent to any server instance with identical expected results, allowing seamless scaling and instance failure tolerance.
Data that must be persisted or shared (sessions, files, databases) should be stored in centralized external stores accessible to all instances.

Common examples:

Sessions: Storing session data in centralized stores like Redis or RADIUS instead of in-memory within each instance avoids authentication errors when requests route to different servers.
File Storage: Uploaded files should be stored in shared object storage (e.g., S3, Cloudflare R2), not locally on individual instance disks.

Key Insight: Designing for statelessness requires careful planning from the codebase to infrastructure to ensure data externalization, facilitating horizontal scaling and fault tolerance.

2. Load Balancers and Their Algorithms

Load balancers (LBs) are critical for distributing incoming client requests across multiple backend instances in horizontal scaling architectures.

All client requests are routed to a load balancer that forwards requests to backend servers based on defined algorithms.
The load balancer hides the complexity of multiple instances from clients, acting as a single access point.

Common Load Balancer Algorithms:

Algorithm	Description	Use Case/Drawbacks
Round Robin	Distributes requests in a fixed rotating order across servers.	Simple but unaware of request complexity or server load; may lead to overload.
Weighted Round Robin	Like round robin but sends more requests to servers with higher capacity based on weights.	Improves distribution if servers have different specs, but still unaware of request type/weight.
Least Connections	Sends new requests to the server with the fewest active connections.	Better at handling variable request loads; balances based on current load.
Weighted Least Connections	Like least connections but considers server capacity weights.	Balances load intelligently considering both load and capacity.
Least Response Time	Routes to servers returning faster responses, prioritizing efficient nodes.	Adapts to server health/performance dynamically.
Resource-Based	Routes based on server resource usage (CPU, RAM).	More complex; reacts to actual system metrics.

Handling Server Failure: Health Checks

Load balancers perform periodic health checks by sending test requests (e.g., simple GET requests expecting HTTP 200).
If a backend instance fails to respond successfully, the LB blacklists it, stopping user traffic routing to the unhealthy server.
Once the server recovers and passes health checks, it is re-added to the LB rotation.
This mechanism prevents user requests from being sent to failed servers, improving reliability.

Key Insight: Load balancer algorithms and health checks are essential for efficient traffic distribution, fault tolerance, and scalable horizontal architectures.

3. Database Scaling: Read Replicas and Sharding

Databases represent the challenging stateful component in backend scaling due to data consistency and storage constraints.

3.1 Read Replicas

One primary (master) database handles write operations (inserts, updates, deletes).
Multiple read replicas (secondary instances) handle read operations (SELECT queries).
Replicas improve read throughput and reduce latency by serving local traffic closer to user regions.

Feature	Description
Write Requests	Handled only by the primary database instance.
Read Requests	Distributed to replicas, reducing load on primary.
Performance Gain	Can reduce primary load to about 30% if ~70% of requests are reads.
Latency Improvement	Replicas placed geographically closer to users for quicker reads.

Major Challenge:

Replication lag: Time delay (ranging ~200ms or more due to physical distances and network delays) between the primary updating data and replicas reflecting those changes.
This can cause read-after-write inconsistency, e.g., a user updating their profile may see outdated info if the read request hits a lagging replica.

Solutions to Replication Lag

Route read queries immediately after writes to the primary to ensure up-to-date data.
Track the replication lag and delay reads from replicas until replication catches up.
Design frontend with planned latency (e.g., delay fetching updated data for a short time).

Modern managed databases (e.g., AWS RDS, Google Cloud SQL) simplify configuring replicas and handling consistency.

3.2 Sharding (Partitioning)

Sharding splits large database tables into smaller, more manageable pieces called shards, distributed across multiple physical database instances.
Each shard contains a subset of data based on a sharding key, e.g., partitioning orders by date range or user ID.

Problem Addressed	Sharding Solution
Query latency on huge tables	Smaller shards reduce rows scanned to speed up queries.
Database instance resource limits	More instances increase total capacity to serve requests.

Example: An orders table partitioned by date range, e.g., January–June orders in one shard, July–December in another.

Key Insight: Sharding balances load and reduces latency but adds complexity in routing queries to the correct shard.

3.3 Distributed Databases

Emerging distributed databases (e.g., PlanetScale, Neon, CockroachDB, Yugabyte) abstract sharding, replication, and consistency complexities.
For most teams, it's recommended to use managed solutions rather than building custom infrastructure due to complexity and maintenance overhead.

4. Content Delivery Networks (CDNs) and Edge Computing

Latency from geographic distance constitutes a major bottleneck unavoidable due to the speed of light limit in fiber optic cables (~200,000 km/s).

A request from Tokyo to a US East (e.g., Virginia) data center involves ~20,000 km roundtrip, imposing a minimum 100ms latency baseline.
Additional backend processing and network hops add up to typical latencies between 500-800ms.

CDN Advantages

Edges or Points of Presence (PoPs) are distributed globally near users (Tokyo, Mumbai, Singapore, etc.).
By caching content at these edges, requests don't need to traverse long distances, reducing latency to 2-3 ms.
CDNs reduce load on origin servers by serving cached content locally, improving scalability.

Typical CDN Cache Content

Static assets: JavaScript, CSS, HTML bundles, images, fonts, videos.
Cacheable API responses: e.g., product catalogs that change infrequently.
Support invalidation or cache purging based on tags or triggers to keep data fresh.

Security Benefits of CDNs

Protection against distributed denial-of-service (DDoS) attacks by absorbing and filtering massive attack traffic.
Globally distributed infrastructure mitigates server crashes and expensive overprovisioning due to malicious traffic surges.

Edge Computing

Extends traditional CDN caching by allowing processing at edge nodes (e.g., running authentication checks, user customization).
Reduces roundtrip times further by executing selected logic near users (e.g., validate session cookies, localization).
Edge functions typically run on resource-constrained environments (limited RAM and CPU).
Examples include Cloudflare Workers using V8 JavaScript isolates.
Not suitable for all backend workloads due to resource and constraint limitations.

Key Insight: CDNs and edge computing significantly improve latency and reduce server load but complement rather than replace central data centers.

5. Asynchronous Processing to Reduce Perceived Latency

Asynchronous processing offloads non-critical, longer-running tasks from synchronous user request paths, improving perceived latency and user experience.

Tasks like sending emails, notifications, video transcoding, image resizing, and bulk deletion are ideal candidates.
Example: Sending an invite email can be offloaded after recording the invite in the database, enabling fast user response (100ms) vs. including slow external email API calls (400ms).

Implementation Pattern

Use message queues (e.g., Redis Queue, RabbitMQ, Kafka) to enqueue background jobs.
Worker processes dequeue tasks and execute them autonomously.
Improves frontend responsiveness and system throughput.

Use Cases

Operation	Reason for Async Processing
Sending emails	Delays due to external service calls are irrelevant to immediate user interaction.
Video/photo processing	Computationally heavy, often minutes-long background job.
Account deletion	Deleting large amounts of user data can take seconds; responding fast improves UX.

Key Insight: Early incorporation of asynchronous design patterns is recommended for better scalability and user experience.

6. Microservices: Organizational and Scaling Benefits vs. Complexity

Monolithic Architecture

Single deployable unit encompassing all backend functionalities.
Simple to develop, test, deploy, and scale horizontally.
Suitable for small to medium teams and simple applications.

Microservices Architecture

Backend decomposed into independent deployable services (e.g., payments, notifications, markdown processing).
Enables independent scaling of components based on resource needs.
Facilitates using different technology stacks/languages tailored for each service.
Primarily beneficial for large teams (100+ developers) to reduce deployment dependencies and improve team productivity.

Advantages of Microservices

Fine-grained scaling by service.
Independent deployment cycles.
Diverse technology choices per service.
Organizational scaling: reduces conflicts across large teams.

Trade-offs and Challenges

Increased network latency due to inter-service communication (network calls replace local function calls).
Handling failure scenarios in distributed systems is more complex.
Management overhead: service discovery, versioning, orchestration.
Complexity in testing and debugging.

Key Insight: Microservices provide major benefits for large-scale, complex systems, mainly at the team and organizational level, but introduce architectural complexity and require mature DevOps practices.

Summary Table of Key Concepts

Concept	Core Principle / Benefit	Challenges / Considerations
Statelessness	Externalize session/state to enable horizontal scaling	Requires architecture redesign and persistent stores
Load Balancer Algorithms	Efficient request distribution and health checks	Choosing appropriate algorithm; handling failures
Read Replicas	Scale reads, reduce load and latency	Replication lag causing data inconsistency
Database Sharding	Partition data for latency reduction and scaling	Complexity in routing and shard management
Content Delivery Networks (CDNs)	Reduce latency by caching near users	Cache invalidation, limited to cacheable content
Edge Computing	Execute logic at the edge for low latency	Resource constraints, limited runtime capabilities
Asynchronous Processing	Offload slow tasks to improve perceived latency	Identifying suitable tasks; managing queues/workers
Microservices	Scale teams, deploy independently, use best tech stack	Added network latency, deployment complexity

This comprehensive overview serves as a foundational guide to understand the principles, architecture patterns, and operational considerations essential for scaling backend systems effectively in modern cloud-driven environments.