Caching Deep Dive (System Design for Beginners – Episode 45)

Caching is a fundamental optimization technique used in high-scale software systems to store frequently accessed data in a faster, temporary storage layer. By introducing a cache between your application and your database, you drastically reduce read latency, scale throughput, and protect your primary storage from expensive, repeated read queries.

1. Why Do We Need Caching? (The Status Quo & Problem Statement)

As a software system scales and handles a growing user base, repeatedly reading data directly from the database becomes a major bottleneck.

• Defining "Expensive": In software engineering, an "expensive" operation does not necessarily refer to monetary cost; rather, it means time. Reading from a database on disk is slow for users when executed repeatedly at high traffic volumes.
• The Disk vs. Memory Bottleneck: Databases typically store data on hard drives or Solid State Drives (SSDs). While disk storage is persistent and cheap, read operations take several milliseconds (e.g., 5 to 10 ms). Conversely, reading from RAM (memory) takes microseconds or fractions of a millisecond (0.1 to 1 ms).
• The Core Trade-off: Caching trades memory (RAM) for speed, yielding dramatic performance improvements that make applications up to 10x faster for cached data reads.

2. Core Mental Model: The Storage Hierarchy & What a Cache Actually Is

To understand caching fundamentally, we must look at how computing hardware organizes storage:

Clarifying Common Misconceptions

• Cache is NOT a new database: A cache is simply a temporary, in-memory copy of data that already exists in the primary database.
• Volatile Nature: Because caches reside in RAM, if the server restarts, the cached data is lost. This is acceptable because the master data remains safely stored in the persistent database.
• Size Constraints: Memory is expensive and physically limited. You cannot store everything in a cache; therefore, you must be selective and cache only the most frequently accessed data.

3. Cache Hits vs. Cache Misses: The Request Flow

When an application receives a request, the server evaluates whether the requested data exists in the cache. This creates two distinct operational paths:

1. Cache Hit: The server looks up the key (e.g., product_id:12345) in the cache, finds the data instantly in memory, and returns it to the client. This is the optimal, high-speed path.
2. Cache Miss: The key is not found in the cache. The server must query the database, retrieve the record, populate the cache with this data for future requests, and then return the response to the client.
   • Note: A cache miss is completely normal and expected; it acts as a self-healing mechanism that populates the cache dynamically on demand.

4. What Kinds of Data Should You Cache?

Caches are most effectively deployed across specific data buckets where access patterns exhibit high repetition and low volatility:

5. Managing Staleness: Time-To-Live (TTL)

Because cached data is a copy, it risks becoming stale if the underlying database record changes. The simplest and most common control mechanism to prevent permanent staleness is TTL (Time-To-Live).

• How TTL Works: When writing an item to the cache, you assign an expiration timer (e.g., 30 seconds for dynamic data, 5 minutes for moderately static data, 1 hour for unchanging configurations).
• Expiration Cycle: Once the TTL expires, the cache automatically deletes the item. The very next read request results in a cache miss, fetches fresh data from the database, and repopulates the cache with a fresh TTL.
• The TTL Balance:
  • TTL too high: Data remains stale for an unacceptably long period.
  • TTL too low: Frequent cache expiration forces constant database hits, negating performance benefits.

6. Read Policies: How Data Enters the Cache

Systems implement specific read strategies to determine how and when data flows into the cache layer.

A. Cache-Aside (Lazy Loading) - Most Common

In this pattern, the application code explicitly manages interactions between the cache and the database.

• Pros: Simple to implement, and only caches data that is actually requested (no wasted memory).

B. Read-Through

In this pattern, the cache sits directly in front of the database, abstracting the database away from the application code.

• Behavior: The application only talks to the cache. If a cache miss occurs, the cache itself transparently fetches the missing data from the database, stores it, and returns it to the application.
• Pros: Simplifies application logic.
• Cons: Provides less granular control compared to Cache-Aside.

7. Write Policies: Maintaining Consistency Between Cache and Database

When data updates occur, maintaining synchronization between the database and the cache becomes critical.

A. Write Around (Cache Invalidation) - Simple & Safe

1. Client sends a write request to update a record (e.g., product price).
2. Server writes the updated data directly to the database.
3. Server deletes (invalidates) the corresponding item from the cache.
4. The next read request encounters a cache miss, fetches the newly updated data from the database, and repopulates the cache.

B. Write-Through

1. Client sends a write request.
2. Server writes the data to the database and simultaneously updates the cache with the new value.

• Pros: Keeps the cache perpetually fresh.
• Cons: Increases write latency because every write triggers two concurrent storage operations.

C. Write-Back (Write-Behind)

1. Server writes data directly to the cache first.
2. The cache asynchronously flushes and writes the data to the database later in the background.

• Pros: Delivers extremely fast write performance.
• High-Risk Caveat: If the cache crashes before the background job writes to the database, data loss occurs. Used only in specialized, high-throughput write scenarios where minor risk can be tolerated.

8. When Should You NOT Use a Cache?

Caching is a powerful optimization, but applying it blindly can degrade performance or waste resources. Avoid caching when:

• Data Changes Constantly: If values update every second (e.g., real-time stock ticker prices), constant cache invalidation causes continuous cache misses and extra overhead.
• Access Patterns are Highly Random: If every user requests unique, non-repeated data, the cache hit rate drops close to zero, adding unnecessary latency by checking the cache first for no benefit.
• The Data Set is Too Large: If the working data set exceeds available RAM capacity, thrashing occurs as items are constantly evicted and reloaded.
• Writes Outweigh Reads: In write-heavy workloads, constant invalidations prevent you from reaping the benefits of cache hits.

Summary and Key Takeaways

• Caching Definition: A temporary, in-memory storage layer designed to store frequently accessed data and accelerate read operations.
• Speed vs. Memory Trade-off: Caching trades valuable RAM for dramatic speed improvements, dropping read latencies from milliseconds (disk) down to microseconds (RAM).
• Cache Hits vs. Misses: Hits serve data instantly from memory; misses query the underlying database and populate the cache for future requests.
• Primary Data Buckets: Ideal candidates for caching include read-heavy static data, user session tokens, and computationally expensive aggregated query results.
• TTL (Time-To-Live): The primary mechanism for balancing data freshness against performance by automatically expiring stale cache entries.
• Read Strategies: Cache-Aside (Lazy Loading) requires application-level coordination, while Read-Through delegates database fetching logic to the cache layer.
• Write Strategies: Write Around (Invalidation) deletes keys upon update; Write-Through updates both storage layers concurrently; Write-Back writes to cache first for speed at the cost of data safety risks.
• Prerequisite for Caching: Caching only succeeds when data access exhibits high repetition and read-heavy traffic patterns.