System Design 13: Design Social Media like Facebook | Instagram | Feeds Generation

This comprehensive study note covers the high-level design (HLD), low-level design (LLD), API contracts, data partitioning, and feed generation mechanisms (Push, Pull, and Fan-out models) for a massive-scale social media platform (similar to Facebook or Instagram) handling 500 million Daily Active Users (DAU).

1. Why Do We Need This? (Problem Statement & Status Quo)

Designing a social media platform like Facebook or Instagram is one of the classic, most frequently asked questions in distributed systems interviews. The fundamental engineering challenge is not merely storing user profiles or uploading media, but orchestrating massive write volumes (posts, likes, comments) and computing real-time personalized timelines (Feeds) for hundreds of millions of users without locking databases or crashing front-end clients.

2. Core Mental Model & Real-Life Analogy

Real-Life Analogy: The Newspaper Delivery Service vs. The Bulletin Board

• The Pull Model (Celebrities / Public Figures): Imagine a famous celebrity posting an update. Instead of printing millions of personal newspapers and hand-delivering them to every subscriber's mailbox instantly (which would waste massive resources), the newspaper sits on a central rack. When a reader walks up to the celebrity's desk, they pull the latest edition.
• The Push Model (Regular Users): Imagine a neighborhood bulletin board. When a regular resident pins a flyer, a postal carrier immediately copies that flyer and drops it into the personal mailboxes of all 50 of their friends so it's waiting for them the moment they open their door.

3. Functional & Non-Functional Requirements

A. Functional Requirements

1. User Onboarding: Users must be able to sign up, log in, view, and update their profile metadata.
2. Post Creation: Users can create and share posts containing text, images, or videos.
3. Social Graph (Following/Friendships): Users can send requests, follow each other, or accept connections (simplified to a unidirectional follow model for uniformity).
4. Interaction: Users can like or comment on posts.
5. Feed Generation: Users can see a personalized feed of posts from the accounts they follow.

Note: Stories, disappearing content, messaging, and chat services are explicitly excluded, as real-time messaging is treated as an independent chat-system design problem.

B. Non-Functional Requirements

• Scale: Massive read and write scale targeting 500 Million Daily Active Users (DAU).
• CAP Theorem Trade-off: The system prioritizes Availability over Consistency ($AP$ system under CAP Theorem), implementing Eventual Consistency. It is acceptable if a post takes a few seconds or a minute to propagate to all followers globally.
• Latency: High upload responsiveness, targeting an average write latency of $\le 500\text{ ms}$ for text/image uploads under normal network conditions.

4. Core Entities Identification

Derived directly from the functional requirements, the core data entities include:

1. User: Credentials, metadata, profile details, and auth tokens.
2. Post: Text content, media URLs, metadata, and timestamps.
3. Follower / Friend: Graph relationship mapping who follows whom.
4. Engagement: Likes, reactions, and threaded comments.
5. Feed: Precomputed or cached timeline references for individual users.

5. API Design & REST Endpoints

A. User Onboarding APIs

B. Post & Feed APIs

C. Interaction APIs

6. High-Level Architecture (HLD)

The high-level architecture implements a microservices-based distributed system fronted by an API Gateway and Load Balancer executing round-robin traffic distribution.

7. Low-Level Design (LLD) & Deep Dive

A. User Service & Schema

The user service manages authentication, registration, and issues JWT tokens upon successful login. Because user metadata is static and relational, a standard relational database is ideal.

CREATE TABLE users (
    user_id UUID PRIMARY KEY,
    username VARCHAR(50) UNIQUE NOT NULL,
    email VARCHAR(100) UNIQUE NOT NULL,
    password_hash VARCHAR(255) NOT NULL,
    phone_number VARCHAR(20),
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

• Database Choice: PostgreSQL / MySQL (ACID compliant, reliable for account data).

B. Content Ingestion Pipeline & Moderation

Due to millions of concurrent writes, writing directly to a database causes bottlenecks. Instead, we use an asynchronous event-driven streaming pipeline powered by Apache Kafka.

Post Database Schema (Cassandra)

Because post writes are write-heavy and append-mostly, Apache Cassandra is chosen for its superior write throughput and distributed partitioning.

CREATE TABLE posts (
    post_id UUID,
    user_id UUID,
    post_type TEXT, -- 'text', 'image', 'video'
    content_text TEXT,
    media_url TEXT,
    thumbnail_url TEXT,
    like_count COUNTER,
    share_count COUNTER,
    comment_count COUNTER,
    created_at TIMESTAMP,
    PRIMARY KEY (user_id, created_at, post_id)
) WITH CLUSTERING ORDER BY (created_at DESC);

C. Follower / Friend Service Schema

Manages directed social graphs (following relationships). A relational database is sufficient unless complex graph recommendations (friends-of-friends) are required.

CREATE TABLE follows (
    follow_id UUID PRIMARY KEY,
    follower_id UUID NOT NULL,
    following_id UUID NOT NULL,
    status VARCHAR(20) DEFAULT 'ACTIVE', -- 'ACTIVE', 'PENDING'
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(follower_id, following_id)
);

D. Engagement Service (Likes & Comments)

High-volume likes and comments are buffered through a Kafka engagement topic before being persisted in a write-optimized NoSQL database (Cassandra) to prevent database lockups.

CREATE TABLE comments (
    comment_id UUID,
    post_id UUID,
    user_id UUID,
    comment_text TEXT,
    like_count COUNTER,
    created_at TIMESTAMP,
    PRIMARY KEY (post_id, created_at, comment_id)
);

CREATE TABLE likes (
    like_id UUID PRIMARY KEY,
    target_id UUID, -- post_id or comment_id
    user_id UUID,
    reaction_type VARCHAR(20),
    created_at TIMESTAMP
);

8. Feed Generation Architecture: Fan-out Models

Computing a user's feed on-the-fly via runtime table joins (INNER JOIN posts ON follows) across thousands of friends will crash the system. Instead, feeds are precomputed using the Fan-out Design Pattern.

A. Push Model (Fan-out-on-Write) vs. Pull Model (Fan-out-on-Read)

B. Hybrid Feed Generation Flow with Caching

To handle both regular users and viral celebrities efficiently, our architecture combines a Fan-out Service, Redis Caches, and Backfilling Workers.

Optimization Caches:

1. Latest Post Cache (Redis Cluster): Stores the 100 most recent posts for every active user, allowing near real-time backfilling without scanning Cassandra.
2. Top Follower Cache (Redis): Caches the subset of friends or celebrities a user interacts with most frequently, preventing unnecessary queries against the entire follower table.

9. Summary and Key Takeaways

• Availability over Consistency: Social media systems favor $AP$ (Availability & Partition Tolerance) under the CAP theorem; eventual consistency for post distribution is completely acceptable.
• Asynchronous Processing with Kafka: High-volume write streams (posts, likes, comments, policy moderation) are buffered via Apache Kafka to guarantee at-least-once delivery and decouple microservices.
• Write-Optimized Databases: Use Cassandra for write-heavy tables (posts, comments) and PostgreSQL for relational metadata (users, direct follow graphs).
• Fan-Out on Write (Push) for Regular Users: Instantly distributes posts to follower feed caches so reads are instantaneous.
• Fan-Out on Read (Pull) for Celebrities: Prevents millions of redundant database writes by computing celebrity timelines dynamically when requested.
• Redis Caching Layer: Precomputes timelines and caches recent user posts to support infinite scrolling and fast feed pagination without overloading core storage.