System Design Interview Prep: Rate Limiter Deep-Dive & Edge Cases

This chapter explores critical follow-up questions commonly asked during system design interviews for a rate limiter. We examine how to handle heterogeneous API limits, scale the storage layer to handle millions of requests per second, and optimize network overhead using local caching strategies.

1. The Challenge of Heterogeneous API Limits

In a naive rate limiter design, we often assume a single, uniform limit (e.g., 100 requests per minute) across the entire system. However, in production environments, a one-size-fits-all limit is highly impractical.

Different API endpoints perform different operations, consume varying levels of system resources, and carry different security risks. For example, reading data is computationally cheap and highly cacheable, whereas creating resources or authenticating users is expensive and sensitive.

2. Rule-Based Rate Limiting

To handle different limits for different APIs, we must move away from hardcoding a single limit in our application code. Instead, we introduce a Rule-Based Rate Limiting mechanism.

Under this approach, the system maintains a dynamic set of rules. When an incoming request arrives, the API Gateway looks up the specific rule matching the target endpoint and applies the corresponding rate-limiting bucket.

API Endpoint Limit Comparison

3. Scaling Storage to 1 Million Requests per Second

If your system scales to handle 1 million API requests per second (RPS), a single Redis instance will no longer suffice.

The Math

• A single, well-optimized Redis instance can handle roughly 100,000 operations per second.
• To support 1,000,000 RPS, you will need approximately 10 Redis instances running in parallel.

Sharding and Partitioning Strategy

Fortunately, rate limiting is a highly partitionable problem. Because User A's rate-limiting bucket is completely independent of User B's bucket, we can cleanly shard our Redis layer without requiring any cross-instance communication.

We can use consistent hashing or a simple modulo hashing algorithm on the identifier (e.g., User ID or IP Address) to route requests to the correct Redis instance.

By ensuring that all requests for a specific user always land on the same Redis instance, we maintain accurate counts without any Redis instances needing to talk to one another.

4. The "Hammering" Problem (Real-World Pitfall)

Even with sharded Redis instances, a major performance bottleneck can occur when a blocked user continuously spams the system. This is known as the Hammering Problem.

The Scenario

1. Alan has a rate limit of 100 requests per minute.
2. Alan quickly exhausts his limit.
3. Alan continues to send spam requests at a rate of 50,000 requests per second.
4. For every single one of those 50,000 malicious requests, the API Gateway must make a network call to Redis to check if Alan's bucket has replenished.

The Impact

Even though the rate limiter successfully blocks Alan from accessing the downstream services, Redis is still kept busy processing 50,000 check operations per second. This unnecessary network and CPU overhead can degrade performance for legitimate users sharing that same Redis shard.

5. Eliminating Network Latency via Local Caching

To solve both the network latency of checking Redis on every request and the Hammering Problem, we can adopt a hybrid caching strategy (similar to the architecture utilized by companies like Stripe).

The Stripe Approach: Local Counters with Asynchronous Syncing

Instead of querying Redis for every single API request, each individual API Gateway instance maintains a small, lightweight counter in its own local, in-memory cache.

1. Local Check: When a request arrives, the API Gateway checks and increments the counter in its local memory (taking virtually 0ms).
2. Asynchronous Synchronization: Every few hundred milliseconds, the local gateway syncs its batch counts with the centralized Redis cluster.
3. Immediate Rejection (Mitigating the Hammering Problem): Once the local gateway receives confirmation that a user is blocked, it caches this "blocked" status locally. Any subsequent spam requests from that user are rejected immediately at the gateway edge, completely bypassing the network call to Redis.

Architectural Trade-offs

Note on Consistency: The trade-off of local caching is loose consistency. During the few hundred milliseconds before the gateway syncs with Redis, a user might slightly exceed their exact rate limit. In most real-world scenarios, this minor discrepancy is an acceptable trade-off for massive gains in system performance and resilience.

6. Complete Architecture Flow Walkthrough

The following workflow demonstrates how an incoming request is processed in a highly scalable, optimized rate-limiting system:

7. Summary and Key Takeaways

• Dynamic Rules over Hardcoding: Do not hardcode rate limits. Use a dynamic rule engine to apply different limits based on the resource cost and security profile of each endpoint (e.g., strict limits on /login, generous limits on GET requests).
• Scale Redis via Sharding: A single Redis instance maxes out around 100,000 ops/sec. Scale to millions of RPS by sharding Redis instances based on a hash of the User ID or IP address.
• Shared-Nothing Sharding: Because rate-limiting buckets are user-specific, Redis shards do not need to communicate with each other, keeping the database layer simple and highly performant.
• Beware the Hammering Problem: Blocked users who continuously spam the system can exhaust database resources if the gateway makes a network call to check their status for every single blocked request.
• Leverage Local Caching: Mitigate network overhead and hammering by caching rate-limit counters and block statuses locally in the API Gateway's memory.
• Embrace Loose Consistency: Accept slight inaccuracies in rate limits (over-limit leakage during sync windows) in exchange for sub-millisecond latency and high system availability.