System Design: Scalable Notification System

This chapter covers the end-to-end system design of a highly available, fault-tolerant, and scalable notification system. Notifications are critical for user engagement and operational updates across modern applications (e.g., payment confirmations, security alerts, and marketing campaigns). We will design a centralized notification service capable of delivering messages across multiple channels while handling massive traffic spikes and ensuring high reliability.

1. Understanding the Problem & Core Requirements

To design an effective notification system, we must first clarify the scope, functional requirements, and non-functional requirements. A centralized notification system must handle different types of notifications with varying delivery urgency and user preferences.

Functional Requirements

• Multi-Channel Support: The system must support four delivery channels:
  1. Push Notifications: Mobile screen alerts (iOS and Android).
  2. SMS: Text messages to mobile phone numbers.
  3. Email: Standard electronic mail.
  4. In-App Notifications: Red notification dots or inbox messages within the application.
• User Preferences: The system must respect user-defined settings. Users must be able to opt out of marketing notifications, though critical transactional notifications (e.g., payment receipts, password resets) cannot be disabled.

Non-Functional Requirements

• High Availability & Scalability: The system must handle massive spikes in traffic (e.g., marketing campaigns sent to millions of users simultaneously).
• Low Latency: Transactional notifications must be delivered within seconds. Marketing notifications can tolerate slight delays.
• Reliability (At-Least-Once Delivery): The system should make every effort not to lose notifications.
• Minimizing Duplicates: While guaranteeing exactly-once delivery in a distributed system is theoretically impossible, the system must minimize duplicate notifications to prevent a poor user experience.

Key Clarifying Questions

When designing this system in an interview, ask the following questions to establish scope:

2. Back-of-the-Envelope Estimation & Scale Challenges

To size our system, we must estimate the daily volume and analyze the performance characteristics of external dependencies.

Scale Estimation

• Daily Volume: 10 million notifications per day.
• Average Throughput:

• Peak Throughput: While $115\text{ QPS}$ is highly manageable for a single database, traffic is rarely distributed evenly. If a marketing campaign targets 1 million users at exactly 9:00 AM, the system will experience a massive, instantaneous spike. The system must be designed to absorb these sudden floods without crashing.

The Challenge of External Dependencies

Our system does not deliver notifications directly to end-user devices. Instead, it acts as an orchestrator that forwards requests to third-party providers:

These external systems introduce three major design constraints:

1. High Latency: Network calls to third-party APIs can take anywhere from $100\text{ ms}$ to over $1\text{ second}$.
2. Unreliability: External providers can experience downtime, rate-limiting, or transient network failures.
3. No Direct Visibility: Once a message is handed off to an SMS or email provider, we generally lose visibility into whether the user actually opened or read the message.

3. API Design & Database Schema

API Design

The system requires a simple, unified endpoint to accept notification requests from various internal microservices (e.g., Order Service, Billing Service).

Endpoint: POST /v1/notify

This endpoint accepts notification requests and immediately returns an acknowledgment.

Request Payload (JSON):

{
  "user_id": "usr_987654321",
  "notification_type": "TRANSACTIONAL",
  "channels": ["PUSH", "SMS", "EMAIL"],
  "message_body": "Your payment of $49.99 was successful.",
  "metadata": {
    "transaction_id": "tx_11223344"
  }
}

Response Payload (JSON):

{
  "status": "accepted",
  "notification_id": "notif_abc123xyz",
  "created_at": "2023-10-27T10:00:00Z"
}

Note: The API returns accepted immediately with a unique notification_id. It does not wait for the actual delivery to complete. This prevents calling services from blocking on slow network operations.

Database Schema

We require a relational database (such as PostgreSQL) to store the persistent state of notifications and user profiles. PostgreSQL can easily handle $100\text{ writes/sec}$ on standard hardware.

Table: notifications

Stores the state and metadata of every notification request.

Table: users

Stores user contact details and device tokens required for delivery.

4. The Naive Approach vs. Asynchronous Architecture

The Naive Approach (Synchronous Processing)

In a naive design, the notification server processes requests synchronously:

Why This Fails at Scale:

• Thread Blockage: The server thread is blocked for $1.5\text{ seconds}$ per request. Under a sudden flood of 1 million requests, the server will quickly run out of threads, leading to connection timeouts and system crashes.
• Cascading Failures: If one external provider (e.g., SendGrid) experiences high latency or downtime, it slows down the entire pipeline, blocking push notifications and SMS deliveries.
• Data Loss on Crash: If the server crashes mid-execution, we lose track of which channels were successfully notified.

The Better Approach (Asynchronous Processing)

To solve these issues, we decouple the ingestion of notification requests from the execution of notification delivery using a message queue.

The Role of Workers

Independent background processes called workers pull tickets from the queue, fetch the necessary details from the database, and handle the slow network calls to external providers.

• Durable Queues: The queue must write messages to disk (not just RAM) so that no tickets are lost if the queue crashes.
• Horizontal Scaling: If the queue size grows during a marketing spike, we can dynamically spin up more worker instances to process messages faster.
• No Single Point of Failure: If a worker crashes while processing a ticket, the queue detects the missing acknowledgment and re-assigns the ticket to another worker.

5. Decoupling Channels: The Multi-Queue Fan-Out Pattern

Using a single queue for all notification channels introduces a major vulnerability: Head-of-Line Blocking. If the email provider becomes slow, email tickets will take longer to process. Workers will get bogged down waiting for email responses, causing push notifications and SMS messages to pile up behind them.

The Solution: One Queue Per Channel

We isolate each delivery channel by creating dedicated queues and worker pools. We use a Fan-Out pattern where the notification server publishes an event, and a copy of that event is routed to each target channel's queue.

Benefits of Channel Isolation:

• Fault Isolation: If the email provider goes down completely, the email queue will back up, but push notifications and SMS messages will continue to deliver instantly.
• Independent Scaling: We can scale worker pools independently based on load (e.g., allocating more workers to the SMS queue during high-priority transactional spikes).

6. Handling In-App Notifications & State Tracking

In-app notifications are unique because they do not rely on external third-party APIs. The delivery and state tracking happen entirely within our own database.

The In-App Workflow

1. Worker Execution: The In-App Worker receives a ticket from the In-App Queue.
2. Database Update: The worker updates the notification record in the database, ensuring the seen flag is set to FALSE.
3. Client Fetching: When the user opens the mobile or web application, the client queries the server for unseen notifications:
   SQL

   Copy

   SELECT * FROM notifications WHERE user_id = 'usr_987654321' AND seen = FALSE;
   

4. State Transition: Once the user opens the notification center and views the alerts, the client sends an update request back to the server:
   SQL

   Copy

   UPDATE notifications SET seen = TRUE WHERE notification_id = 'notif_abc123xyz';
   

Note: Unlike SMS and email—where we have no visibility post-dispatch—in-app notifications allow us to track the exact lifecycle of a message from delivery to consumption.

7. Resiliency & Fault Tolerance: Exponential Backoff & Dead Letter Queues

When calling external APIs, transient network failures and rate limits are inevitable. Our workers must handle these failures gracefully.

Exponential Backoff

If a call to an external provider fails, retrying immediately can overload the struggling provider and waste system resources. Instead, workers must retry politely using Exponential Backoff, doubling the delay between each subsequent attempt.

Dead Letter Queues (DLQ)

We cannot retry indefinitely. If a notification fails to deliver after a set number of attempts (e.g., 5 retries), the worker stops retrying and moves the ticket to a Dead Letter Queue (DLQ).

The DLQ acts as a holding area for failed messages. Support engineers can inspect these messages, diagnose the underlying issue (e.g., invalid phone numbers or expired API keys), and safely re-queue them once resolved.

8. The Distributed Systems Trap: Preventing Duplicate Notifications (Idempotency)

In distributed systems, guaranteeing exactly-once delivery is impossible due to network partitions and system crashes. Instead, systems operate on at-least-once delivery, which introduces the risk of duplicate notifications.

The Duplicate Scenario (Crash-After-Send)

Consider this failure sequence:

Because the queue never received an acknowledgment, it assumes the worker failed to process the message. The queue then hands the same ticket to another worker, resulting in the user receiving the same SMS twice.

The Solution: Idempotency Keys with Redis

To prevent duplicates, we must make the delivery process idempotent. We use the unique notification_id as an Idempotency Key and store it in a fast, shared cache (Redis) with a Time-To-Live (TTL) of 1 minute.

Step-by-Step Idempotent Execution Flow:

Why Use a TTL (Time-To-Live)?

The Redis entry is not permanent. If a worker writes the key to Redis but crashes before calling the provider, the notification would be blocked forever. By setting a 1-minute TTL, the key automatically expires, allowing a subsequent worker to pick up the ticket and safely attempt delivery.

9. Solving the Dual-Write Problem: Database-First Queueing (Outbox Pattern)

In our high-level design, the notification server performs two separate writes:

1. Writes the notification record to the Database.
2. Publishes the ticket to the Queue.

This is known as the Dual-Write Problem. If the server writes to the database but crashes before writing to the queue, the notification is saved but will never be sent.

The Solution: Database-First Queueing (Transactional Outbox Pattern)

To guarantee consistency, we make the database our single source of truth. The notification server only writes to the database.

We then run a lightweight Background Polling Job (or use Change Data Capture) that continuously scans the database for pending records and safely pushes them to the queues.

If the background job pushes a duplicate ticket to the queue due to a network hiccup, our downstream Redis idempotency layer will safely catch and discard it.

10. Real-World Implementations: LinkedIn & Uber Case Studies

To understand how these patterns operate at massive scale, we can look at how major tech companies structured their notification pipelines.

LinkedIn: Air Traffic Controller (ATC)

Historically, every product team at LinkedIn sent notifications independently. This resulted in users being bombarded with redundant emails and push notifications, leading to high opt-out rates.

To solve this, LinkedIn built a centralized platform called Air Traffic Controller (ATC):

• Centralized Gateway: All notification requests flow through a single gateway powered by Kafka.
• Intelligence Layer: ATC analyzes user activity and preferences to decide if, when, and on which channel a notification should be delivered.
• Results: LinkedIn reduced overall email volume by 50%, resulting in a 65% drop in user complaints while maintaining high engagement.

Uber: Priority-Based Delivery

Uber's notification system handles highly time-sensitive updates (e.g., "Your driver has arrived") alongside promotional marketing campaigns.

• Priority Buckets: Uber categorizes notifications into distinct priority levels.
• Throttling & Preemption: High-priority notifications (trip updates) bypass marketing queues entirely. If the system experiences heavy load, promotional notifications are throttled or paused to ensure critical operational alerts are delivered with sub-second latency.

Summary and Key Takeaways

• Asynchronous Decoupling: Never call external notification APIs synchronously. Accept the request, write it to a database, return an immediate acknowledgment, and process delivery asynchronously using queues and workers.
• Channel Isolation: Use dedicated queues and worker pools for each delivery channel (Push, SMS, Email, In-App). This prevents a slow or down provider on one channel from blocking delivery on other channels.
• At-Least-Once Delivery: Exactly-once delivery is impossible in distributed systems. Design for at-least-once delivery and make the processing pipeline idempotent.
• Idempotency with Redis: Use a unique notification_id as an idempotency key. Check and set this key in Redis with a TTL before calling external providers to prevent duplicate deliveries.
• Resiliency Patterns: Implement exponential backoff for retries to avoid overloading external APIs, and route persistently failing notifications to a Dead Letter Queue (DLQ) for manual inspection.
• Outbox Pattern: Avoid the dual-write problem by writing notification requests strictly to the database first, using a background job to reliably publish those records to the message queues.