System Design: Designing a Scalable News Feed

A news feed is a dynamic, constantly updating list of posts, images, and videos displayed on a user's homepage when they open a social media application (such as LinkedIn or X/Twitter). The core technical challenge of a news feed system is to aggregate, sort, and deliver content from followed accounts to millions of active users with sub-second latency. This design explores how to transition from a naive query-on-demand model to a highly optimized hybrid push/pull architecture capable of handling massive scale.

1. Clarifying Requirements & System Scope

To design an effective system, we must first establish the functional boundaries and non-functional performance targets.

Functional Requirements

• Follow Graph: Users must be able to follow other accounts.
• Post Creation: Users can create posts containing text, images, or videos.
• Feed Retrieval: Users can read a feed containing the newest posts from accounts they follow, sorted in reverse chronological order (newest first).
• Feed Pagination: The system should return a set number of posts (e.g., top 50) per request.

Non-Functional Requirements

• Low Latency Feed Generation: The feed must load in under 1 second when a user opens the app or refreshes.
• Propagation Delay: A newly created post should appear in followers' feeds within a few seconds (minutes is unacceptable).
• High Availability: The system must remain highly available, even during peak traffic periods.
• Consistency vs. Performance: Occasional duplicate posts or minor delays in feed updates during a refresh are acceptable trade-offs to ensure low latency.

2. Back-of-the-Envelope Estimation

To understand the scale of our system, we must calculate the read/write ratios and storage requirements based on a target of 10 million Daily Active Users (DAU).

Read Traffic (Feed Loads)

• Daily Active Users (DAU): $10,000,000$
• Average Feed Opens per User per Day: $5$
• Total Feed Reads per Day: $10,000,000 \times 5 = 50,000,000$ reads/day
• Average Read QPS (Queries Per Second):

Write Traffic (Post Creation)

• Average Posts per User per Month: $3$
• Total Posts per Month: $10,000,000 \times 3 = 30,000,000$ posts/month
• Total Posts per Day:

• Average Write QPS:

The Read-to-Write Ratio

Key Insight: The system is highly read-heavy. For every single post written to the platform, there are 50 feed reads. Therefore, our architectural optimizations must prioritize the feed reading path to guarantee low latency.

3. API Design & Data Contracts

The mobile client interacts with our backend servers using simple, stateless RESTful APIs. We will focus on the two core endpoints required for this system.

1. Create a New Post

• Endpoint: POST /v1/posts
• Description: Called by the client application when a user publishes a new post.
• Request Payload (JSON):
  JSON

  Copy

  {
    "author_id": "usr_987654321",
    "content": "Hello world! Check out my new photo.",
    "media_url": "https://cdn.myplatform.com/media/img_12345.jpg"
  }
  

• Response (JSON):
  JSON

  Copy

  {
    "post_id": "post_11223344",
    "status": "success",
    "created_at": 1718294400
  }
  

2. Retrieve the News Feed

• Endpoint: GET /v1/feed
• Description: Called by the client application to load the user's personalized news feed.
• Query Parameters:
  • user_id (string, required): The ID of the user requesting the feed.
  • limit (integer, optional): Number of posts to return (default: 50).
• Response Payload (JSON):
  JSON

  Copy

  {
    "posts": [
      {
        "post_id": "post_11223344",
        "author_id": "usr_987654321",
        "content": "Hello world! Check out my new photo.",
        "media_url": "https://cdn.myplatform.com/media/img_12345.jpg",
        "timestamp": 1718294400
      }
    ],
    "next_cursor": "post_11223300"
  }
  

4. Database Schema & Access Patterns

To support efficient queries, we define three primary database tables.

1. Users Table

Stores user profile information. It includes a critical boolean flag is_celebrity to optimize feed generation.

2. Follows Table

Represents the social graph. This is a highly queried table with a many-to-many relationship.

Scale Note: If 10 million users follow an average of 200 accounts each, this table will contain 2 billion rows. It must support fast lookups in both directions:

1. Who does Alan follow? (Used to pull posts)
2. Who follows John? (Used to push posts)

3. Posts Table

Stores the metadata and text content of all posts.

5. The Naive Approach: Fan-Out on Read (Pull Model)

In the simplest implementation, no work is done beforehand. The system builds a user's feed dynamically at the exact moment they open the application.

Step-by-Step Execution Flow

Why This Naive Approach Fails at Scale

While simple, this approach places an unsustainable load on the database as the user base grows:

1. High Database Read Amplification:
   • At an average of 600 feed opens per second, the server must query the follows table 600 times/sec.
   • If each user follows 300 accounts, the server must execute $600 \times 300 = \mathbf{180,000}$ database lookups every single second to fetch recent posts.
   • During peak traffic (e.g., $5 \times$ average), this spikes to 900,000 lookups per second, crashing the database.
2. Wasted Computation:
   • If a user opens the app, closes it, and reopens it 10 seconds later, the system repeats the entire process of querying, merging, and sorting, even if none of the followed accounts have posted anything new.

6. The Precomputation Approach: Fan-Out on Write (Push Model)

To achieve sub-second feed load times, we shift the heavy lifting from the read path to the write path. Instead of building the feed when a user reads it, we precompute and store each user's feed in a fast, in-memory cache.

The Two-Cache Architecture

We deploy two distinct Redis cache layers to handle different parts of the feed:

1. Feed Cache (Redis Lists): Stores a mapping of user_id to a list of post_ids. It does not store actual post content, only references.
2. Post Cache (Redis Key-Value): Stores the actual post content (author, text, media URLs) mapped to the post_id.

Step-by-Step Precomputation Flow (Push Model)

When a user (e.g., John) publishes a new post, the system pushes it directly into the feed lists of all his followers.

How the Feed is Read

When Alan opens his app, the read path is incredibly fast:

Note: If a post is missing from the Post Cache (a cache miss), the server queries the database, populates the Post Cache for future requests, and returns the post.

7. Deep Dive: Reliable Event Delivery via the Outbox Pattern

In the precomputation model, saving a post to the database and pushing it to the message queue are two separate operations. If the server crashes after saving the post but before queuing the background job, the database and the cache will become permanently out of sync (followers will never see the post).

To solve this, we use the Transactional Outbox Pattern.

The Outbox Workflow

Instead of writing to the database and queue separately, we write to a posts table and an outbox table within the same database transaction. This guarantees that either both writes succeed, or both fail.

Background Processing Pipeline

Database (Outbox Table)
   |
   | 1. Polls PENDING jobs
   v
Outbox Publisher (Service)
   |
   | 2. Publishes ticket to Message Queue
   v
Message Queue (e.g., RabbitMQ / Kafka)
   |
   | 3. Updates Outbox status to DONE
   v
Database (Outbox Table)
   |
   | 4. Consumes ticket
   v
Worker Node (Fan-out Service)
   |
   | 5. Queries Follows Table
   v
Database (Follows Table)
   |
   | 6. Pushes post_id to followers' lists
   v
Feed Cache (Redis)

Ensuring Idempotency

If a worker crashes after pushing a post to Redis but before acknowledging the ticket in the Message Queue, another worker will reprocess the same ticket. This could result in duplicate posts in a user's feed.

To prevent this, the worker performs an idempotency check before inserting:

# Pseudo-code for idempotent worker insertion
def push_to_feed(follower_id, post_id):
    # Check if post_id already exists in the user's Redis list
    if not redis.list_contains(follower_id, post_id):
        redis.lpush(follower_id, post_id)
        # Trim the list to keep only the latest 500 posts
        redis.ltrim(follower_id, 0, 499)

8. The Celebrity Problem & The Hybrid Solution

While the precomputation (push) model works perfectly for normal users, it breaks down completely when applied to celebrity accounts.

The "Messi" Problem (Write Bottleneck)

If Lionel Messi has 100 million followers and publishes a post:

• The fan-out worker must write his post_id to 100 million separate Redis lists.
• This single write operation takes minutes to complete, overloading the Redis cluster and delaying updates for other users.
• Many of those 100 million followers may be inactive users who haven't logged in for months, meaning the system is wasting computation and memory precomputing feeds they will never see.

The Hybrid Solution: Push for Normal Users, Pull for Celebrities

To solve this, we classify users dynamically based on their follower count:

• Normal Users (< 100,000 followers): Use Fan-out on Write (Push).
• Celebrity Users (>= 100,000 followers): Use Fan-out on Read (Pull).

Merging the Two Streams on Read

When Alan opens his app, the system dynamically merges his precomputed feed with real-time pulls from the celebrities he follows:

9. Media Storage & Content Delivery Networks (CDNs)

Storing raw images and videos directly inside database tables or Redis caches is highly inefficient. It bloats database sizes, slows down backups, and degrades cache performance.

Media Architecture

Instead of storing raw binary data in the database, we store media files in dedicated object storage and reference them via URLs.

Content Delivery Network (CDN) Integration

To ensure fast media loading times globally, we place a CDN in front of our object storage.

10. Scaling, Sharding, and Handling Viral Posts (Hotkeys)

As the platform grows to support hundreds of millions of posts, we must address database scaling and cache hotspots.

Database Sharding (Cassandra for Posts)

The posts table grows indefinitely (e.g., 360 million posts per year). A single database instance cannot store or index this volume of data.

• Database Choice: Apache Cassandra is ideal for the posts table because it is optimized for time-ordered, write-heavy data and does not require complex joins.
• Sharding Key: We shard the posts table using a composite key: author_id as the partition key and timestamp as the clustering key. This ensures all posts by a specific user are stored together and sorted chronologically on disk.

Cache Sizing Calculations

Let's calculate the memory footprint required for our Redis clusters.

1. Feed Cache (Redis Lists)

• Active Users: 10 million (we only maintain precomputed feeds for users active within the last 30 days).
• Max Posts per Feed: 500 post IDs.
• Size of Post ID: 8 bytes.
• Memory per User: $500 \times 8 \text{ bytes} = 4,000 \text{ bytes (4 KB)}$.
• Total Memory Required:

2. Post Cache (Redis Key-Value)

• Posts per Month: 30 million.
• Average Post Metadata Size: 1 KB (text, author ID, media URLs).
• Total Memory for 1 Month of Posts:

Both caches can easily fit within standard Redis cluster configurations.

Handling Viral Posts (The Hotkey Problem)

When a post goes viral, millions of users read the exact same post simultaneously. This causes a "hotkey" issue, where the single Redis node storing that specific post ID is overwhelmed by read requests.

Mitigation: Cache Replication & Local Caching

To prevent cache nodes from crashing under viral load:

1. Read Replicas: We replicate the hot post across multiple Redis read replicas and distribute read queries across them.
2. In-Memory Client Caching: The web servers can cache highly active posts in their local memory (e.g., Guava cache) for a few seconds, preventing the read requests from reaching the Redis cluster at all.

11. Summary and Key Takeaways

• Read-Heavy Nature: A news feed is a highly read-heavy system (50:1 read-to-write ratio), requiring optimizations that favor fast read paths.
• Fan-out on Read (Pull): Simple to implement but causes massive database read amplification under high user traffic.
• Fan-out on Write (Push): Precomputes feeds in Redis lists, enabling sub-second load times at the cost of higher write complexity.
• The Hybrid Model: The optimal industry standard. Normal users have their feeds pushed, while celebrity posts are pulled dynamically at read time to avoid write bottlenecks.
• Transactional Outbox Pattern: Ensures reliable, eventual consistency between the database and the precomputed Redis feeds, preventing lost updates during server crashes.
• Media Separation: Heavy media files (images/videos) are stored in object storage and served via CDNs, keeping database and cache payloads lightweight.
• Active User Optimization: Feeds are only precomputed for active users (e.g., active within the last 30 days) to save memory and CPU cycles. Feeds for inactive users are rebuilt on demand when they log back in.