Caching in Distributed Systems: A Friendly Introduction

Caching is a foundational pillar of computer science and large-scale distributed systems. Almost every high-performance system incorporates caching across multiple tiers—from client mobile devices to backend server networks and databases—to dramatically reduce latency, eliminate redundant computations, and protect upstream databases from catastrophic load spikes.

1. Why Do We Need Caching? (The Latency Problem)

In modern web applications, every user request triggers a cascade of network calls and computational workloads. Consider a classic user request for an Instagram newsfeed:

CODE

Client (User App)
   |
   | (100 ms network latency)
   v
Application Server
   |
   | (10 ms query execution)
   v
Database Server (Source of Truth)

Complete Round-Trip Breakdown

Client to Server: 100 ms network transit time.
Server to Database: 10 ms query execution and transport.
Database to Server: 10 ms response payload return.
Server to Client: 100 ms final response transmission.

Total End-to-End Latency: 220 ms

While optimizing client-to-server network latency requires specialized architecture (like CDNs), backend engineers focus heavily on optimizing the server-to-database leg. If a database query takes 10 ms, multiplying that across thousands of concurrent services creates massive bottlenecks. Caching solves this by serving frequent data out of lightning-fast local or distributed memory, cutting query times down from milliseconds to microseconds.

2. Core Mental Model: Reducing Repeatable Work Through Storage

Real-Life Analogy: The Librarian and the Frequent Request

Analogy: Imagine working at a massive municipal library. When a patron asks for a rare, obscure 18th-century manuscript, you must walk down to the deep archives, search through miles of shelving, and carry it back—a process taking 15 minutes.

However, if 50 people ask for that exact same manuscript throughout the day, walking to the archive every single time is wasteful. Instead, you keep that manuscript right on your front desk. When the next person asks, you hand it to them in 2 seconds. That front desk is your Cache.

Cohort grouping exemplifies this principle in software. For example, a young software engineer in India who loves football will consume a newsfeed very similar to another young software engineer in India with identical interests.

Instead of recomputing the database query for every user, the system generates the feed once, stores it in fast memory, and serves subsequent identical or cohort-matched requests instantly:

Structure & Concept Flow

Incoming Request (User: Cohort A)

Check Local/Distributed Memory Cache

+---> [Cache HIT] ---> Return Data in 1 ms (No DB call!)

+---> [Cache MISS] ---> Query Database (10 ms) -> Save to Cache -> Return Response

3. Naive vs. Production Approach: The Memory Constraint Dilemma

A naive engineer might look at caching and ask: "If caching is so fast, why don't we take our entire multi-terabyte database and load it all into memory?"

Why Complete Database Caching Fails at Scale

Cost: RAM (Random Access Memory) is significantly more expensive per gigabyte than SSD/HDD storage. Fitting terabytes or petabytes of data into RAM is financially prohibitive.
Data Distribution: Real-world workloads follow power-law distributions (like Zipfian distribution). A small fraction of data (e.g., 10-20% of posts or user profiles) accounts for 80-90% of all user requests.

Therefore, production systems use selective caching: storing only the most frequently accessed subset of data in memory and falling back to the database for the long tail of unpopular queries.

4. The Two Fundamental Questions of Caching

Whenever you introduce a cache between your application and your database, you must answer two architectural questions:

Write Policies (Cache Invalidation & Updates): When data changes in the database (the source of truth), how do you keep the cache synchronized? Do you update them together, invalidate the cache entry, or update asynchronously?
Eviction Policies: When your cache memory fills up and a new viral video or popular record needs to be loaded, what existing data do you kick out?

5. Real-World Practical Pitfalls: Thrashing and Invalidation

Caching is not a silver bullet; poorly configured caches can actively degrade system performance.

Pitfall 1: Cache Thrashing (Poor Hit Rates)

When your cache is too small to hold the working set of data, or when clients query data in strict sequence, you encounter thrashing—a cycle of constant evictions and database re-fetches that increases latency instead of reducing it.

Step-by-Step Thrashing Walkthrough

Capacity: Cache can hold a maximum of 3 elements.
Client Request Sequence: 1, 2, 3, 4, 1, 2...

Structure & Concept Flow

Request 1 -> Cache Missing -> Load 1 from DB. Cache: [1]

Request 2 -> Cache Missing -> Load 2 from DB. Cache: [1, 2]

Request 3 -> Cache Missing -> Load 3 from DB. Cache: [1, 2, 3]

Request 4 -> Cache Full! Evict LRU (1). Load 4 from DB. Cache: [2, 3, 4]

Request 1 -> Cache Missing! Evict LRU (2). Load 1 from DB. Cache: [3, 4, 1]

Request 2 -> Cache Missing! Evict LRU (3). Load 2 from DB. Cache: [4, 1, 2]

Result: Every single request results in a Cache Miss, forcing database queries plus the overhead of constant cache population and eviction. Latency spikes significantly.

Pitfall 2: Eventual Consistency and Stale Data

Because the cache holds a copy of data rather than the primary source of truth (the database), there is always a lag before updates propagate.

Note: For non-critical data—such as YouTube video like counts—serving stale data for a few minutes is acceptable. However, for financial transactions or inventory systems, stale cache data can lead to overselling items or showing incorrect account balances. Systems must manage this trade-off using eventual consistency models.

6. Where Should You Place Your Cache?

Choosing where to place your cache determines how your system scales, how easily it can be maintained, and how much memory overhead your application servers endure.

Cache Placement Strategy	Architecture & Location	Pros	Cons
1. In-Memory Cache	Embedded directly inside the application server process (e.g., Java `HashMap`, Guava Cache).	Zero network hop; ultra-low latency (microseconds).	Memory is bound to the app server; cannot be shared across multiple service instances easily; risk of Out-Of-Memory (OOM) crashes.
2. Database-Internal Cache	Built directly into the database engine as a black box (e.g., buffer pool in PostgreSQL/MySQL).	Automatic management; requires zero application code changes.	Limited control by application engineers; scoped strictly to that specific database instance.
3. Distributed Global Cache	Standalone, external cluster of cache servers (e.g., Redis, Memcached) queried via network APIs.	Highly scalable independently; shared across multiple microservices; language-agnostic.	Incurs a small network round-trip cost (though still faster than a disk-based database query).

In modern large-scale production architectures, all three strategies are used simultaneously, with the Distributed Cache serving as the primary workhorse for microservices.

7. Summary and Key Takeaways

Fundamental Purpose: Caching reduces repeatable computational work and database load by storing frequently accessed data in faster, closer memory tiers.
Latency Impact: By bypassing slow database round-trips (10 ms+), caches drop read response times down to 1 ms or less.
Selective Storage: Because memory is expensive and limited, systems cannot cache everything; engineers must predict access patterns and cache only popular data.
Cache Eviction Policies: When memory fills up, algorithms like Least Recently Used (LRU) or Least Frequently Used (LFU) determine which items to discard.
Write Consistency Challenges: Maintaining copies creates consistency gaps; systems must balance freshness (strong consistency) against performance (eventual consistency).
Thrashing Risks: Sequential scanning on undersized caches causes high miss rates and wasted overhead, slowing the system down instead of speeding it up.
Strategic Placement: While in-memory and database caches have niche benefits, distributed caches (like Redis) are the industry standard for large-scale distributed systems due to independent scalability and shared access across microservices.