In-Memory Scaling: Standalone Redis Architecture

To scale an application to handle massive traffic, relying solely on disk-bound databases eventually becomes a major bottleneck. When a relational database like PostgreSQL reaches its physical limits, scaling up hardware becomes prohibitively expensive. Moving active datasets entirely into memory using Redis offers a high-performance, cost-effective alternative.

1. The Bottleneck of Disk-Bound Databases

As database size grows, disk I/O operations become the primary bottleneck for high-throughput applications.

The Status Quo

• The Dataset: A PostgreSQL database containing 11 million records (specifically a codes table used for a URL shortener application) occupies approximately 16 GB of storage on disk.
• The Performance Ceiling:
  • Writes: PostgreSQL write throughput maxes out at approximately 40,000 to 50,000 requests per second (RPS).
  • Reads: PostgreSQL read throughput tops out at around 500,000 RPS under heavy load, but scaling this further requires expensive database instances, read replicas, and complex connection pooling.
• The Financial Cost: Attempting to scale a disk-bound relational database to handle these loads can easily cost upwards of $30,000 per month in cloud infrastructure.

2. The In-Memory Mental Model

Instead of querying a disk-bound database, you can load the entire active dataset into Random Access Memory (RAM).

Real-Life Analogy

Think of a disk-bound database as a massive archive warehouse down the street. Every time you need a file (a database record), a runner has to walk to the warehouse, search the shelves, and bring it back to your desk.

Moving your database to Redis is like taking the most frequently used filing cabinets out of that warehouse and placing them directly on your desk. Now, you can reach out and grab any file instantly without waiting on transit times.

If your host machine has sufficient RAM (for example, a high-spec bare-metal or cloud instance with 256 GB of RAM), a 16 GB database can easily fit entirely in memory with room to spare.

3. Migrating Data from PostgreSQL to Redis

To transition from disk-bound reads to in-memory reads, you must migrate the existing dataset. This is achieved using a batch migration script that reads records from PostgreSQL and writes them to Redis in structured batches.

The Migration Strategy

1. Flush the Redis Cache: Clear out any stale data to ensure a clean slate.
2. Batch Querying: Query PostgreSQL using pagination or batching (e.g., chunks of 2,000 records) to prevent memory exhaustion on the application node.
3. Data Structure Mapping: Map relational rows to Redis data structures.
   • Individual records are stored as Redis Hashes (HSET) for fast field-level lookups.
   • Uniqueness is enforced using a Redis Set (SADD) containing all active IDs.
   • A dedicated key tracks the auto-incrementing sequence for generating new IDs.

Realistic Migration Script (Node.js / TypeScript)

import { Client } from 'pg';
import Redis from 'ioredis';

const pgClient = new Client({ connectionString: process.env.DATABASE_URL });
const redis = new Redis(process.env.REDIS_URL || 'redis://127.0.0.1:6379');

async function migrateData() {
  await pgClient.connect();
  console.log('Connected to PostgreSQL. Flushing Redis...');
  await redis.flushall();

  const batchSize = 2000;
  let offset = 0;
  let hasMore = true;
  let totalMigrated = 0;

  // Track the highest ID to set the auto-increment counter in Redis
  let maxId = 0;

  while (hasMore) {
    // Fetch batch from PostgreSQL
    const res = await pgClient.query(
      'SELECT id, code, created_at FROM codes ORDER BY id LIMIT $1 OFFSET $2',
      [batchSize, offset]
    );

    if (res.rows.length === 0) {
      hasMore = false;
      break;
    }

    const pipeline = redis.pipeline();

    for (const row of res.rows) {
      const { id, code, created_at } = row;
      if (id > maxId) maxId = id;

      // 1. Store the main record as a Hash
      pipeline.hset(`code:${id}`, {
        id: id.toString(),
        code: code,
        createdAt: new Date(created_at).toISOString()
      });

      // 2. Add the code to a Set to enforce uniqueness checks in-memory
      pipeline.sadd('codes:unique', code);
    }

    // Execute batch write in Redis
    await pipeline.exec();
    
    totalMigrated += res.rows.length;
    offset += batchSize;
    console.log(`Migrated ${totalMigrated} records...`);
  }

  // 3. Store the last used ID sequence
  await redis.set('codes:sequence', maxId);

  console.log(`Migration complete! Total records migrated: ${totalMigrated}`);
  await pgClient.end();
  await redis.quit();
}

migrateData().catch(console.error);

4. Standalone Redis Performance Benchmarks

Once the 16 GB PostgreSQL database is migrated, the memory footprint in Redis increases slightly to around 20 GB. This overhead is normal and is due to Redis's internal data structures, key indexing, and tracking sets (like codes:unique).

With the dataset fully loaded into RAM, we run load tests against the Redis-backed endpoints.

Write Performance (Post Request to /code-fast)

• The Logic: The application generates a new short code, checks the codes:unique Redis Set to ensure it does not already exist, writes the new hash to Redis, and increments the sequence counter.
• CPU Utilization: CPU usage hovers around 80%, meaning the system is processing requests actively rather than sitting idle waiting on disk I/O.
• Throughput: ~100,000 RPS.
• Comparison: This is a 2x to 2.5x improvement over PostgreSQL writes (~40,000 - 50,000 RPS).

Read Performance (Get Request to /code-fast)

• The Logic: The application fetches a code directly from Redis using HGETALL code:<id>.
• Throughput: ~300,000 RPS.
• Comparison: While raw read performance is comparable to a highly optimized PostgreSQL read path, it completely bypasses the database. This protects the primary transactional database from read exhaustion and eliminates expensive database scaling costs.

5. The Standalone Redis Limit

While 100,000 writes/sec and 300,000 reads/sec are impressive, we are still far from the target of 1 million requests per second.

Why Standalone Redis Bottlenecks

Redis is fundamentally a single-threaded event loop application.

• It uses a single CPU core to process commands sequentially.
• Even if your host machine has 64 CPU cores and 256 GB of RAM, a single instance of Redis will only utilize one core to execute commands.
• A single Redis instance physically bottlenecks at approximately 100,000 to 150,000 operations per second due to single-threaded CPU limits and network I/O overhead on that single port.

To break past this barrier and utilize the remaining idle CPU cores on our high-powered machine, we must transition from a standalone Redis architecture to a Redis Cluster architecture.

6. Architectural Comparison

Summary and Key Takeaways

• Disk I/O is the Enemy of Scale: Relational databases like PostgreSQL bottleneck on disk writes, capping performance at around 40k-50k RPS under heavy write loads.
• In-Memory Migration: Moving active tables (like URL shortener codes) into RAM allows you to bypass disk bottlenecks entirely.
• Redis Data Structures: Use Redis Hashes (HSET) for record storage, Sets (SADD) for fast uniqueness checks, and Strings (SET/INCR) for sequence tracking.
• The Single-Threaded Ceiling: Standalone Redis is bound by a single-threaded event loop. A single instance caps out at roughly 100k writes/sec, leaving multi-core CPU capacity underutilized.
• Cost Efficiency: Shifting heavy read/write routes to an in-memory architecture can reduce cloud database infrastructure costs by up to 66% (e.g., cutting a $30,000 monthly bill down to$10,000).
• The Path to 1 Million RPS: To scale past the 100k-300k RPS limit of standalone Redis, you must implement Redis Clustering to distribute the load across multiple Redis instances running on separate CPU cores.