✅ System Design: Scalable Notifications System (SMS, OTP, Email & Push)

A notification system is a foundational pillar of modern distributed architectures. It acts as a centralized engine allowing internal or third-party client applications (such as Amazon, Flipkart, or Uber) to dispatch millions of user-facing messages across multiple channels—including Email, SMS, In-App Push, and Bank OTPs—reliably, securely, and with predictable latency.

1. Why Do We Need a Centralized Notification System?

In early-stage or monolithic applications, individual services (e.g., Order Service, Chat Service, Payment Service) often wire up SMTP clients or SMS gateways directly inside their codebase.

The Naive Approach (Direct Third-Party Integration)

Why This Fails at Scale:

• Code Duplication: Every microservice implements boilerplate connection retry logic, credential management, and payload templating.
• Vendor Lock-In: Swapping an SMS provider (e.g., from Twilio to MSG91) requires modifying code across dozens of independent services.
• No Global Rate Limiting or Circuit Breaking: If an external provider goes down, dependent business microservices freeze or crash due to thread exhaustion.
• Lack of Unified User Preferences: Users have no single dashboard to control whether they want emails vs. push notifications across the entire platform.

2. Core Mental Model & Real-Life Analogy

Real-Life Analogy (The Central Mailroom & Postal Service): Think of our Notification System as a Centralized Corporate Mailroom. Individual departments (Clients like Flipkart or Uber) don't personally drive out to drop letters in mailboxes. Instead, they drop off pre-printed templates and address lists at the mailroom reception desk. The mailroom sorts the mail by urgency (Standard Promotional Flyers vs. Urgent Legal/Medical Notices), picks the right delivery vehicle (Email truck, SMS motorcycle, or Hand-delivered Push note), tracks delivery statuses, and honors individual "Do Not Disturb" registry preferences.

3. Functional and Non-Functional Requirements

Functional Requirements

1. Multi-Channel Support: Deliver notifications via Email, SMS, and In-App Push.
2. Delivery Modes: Support both Real-Time notifications (e.g., Bank OTPs) and Scheduled notifications (e.g., promotional marketing blasts).
3. Template Management: Provide clients with dynamic template configuration (supporting persona-based variable substitution like replacing product names or user names).
4. Delivery Status Dashboard: Provide clients with real-time visibility into message delivery states (Pending, Sent, Delivered, Failed).
5. User Preferences: Allow end-users to manage and customize their notification channel preferences (e.g., opt out of SMS marketing while retaining Email alerts).

Non-Functional Requirements

1. High Scale: Handle massive throughput of 1 million notifications per minute.
2. CAP Theorem Priority (AP over CP): Prioritize Availability over strict consistency. If a user updates their template or preference, a slight replication delay (eventual consistency of a few seconds) is fully acceptable, but the service must never reject inbound requests.
3. Tiered Latency SLAs:
   • Critical / OTP Notifications: Near real-time delivery SLA (< 1 to 2 seconds).
   • Promotional / Standard Notifications: Acceptable background delay of 5 to 10 seconds.
4. Message Durability: Zero message loss guarantee once accepted by the notification engine.

4. Core Entities & Database Schema Architecture

Before diving into high-level and low-level designs, we must define our core entities. Note the crucial distinction between a Client (an organization/application like Uber or Flipkart) and a User (the individual human using that application).

Core Entities Table

5. API Design & REST Endpoints

Our public-facing API Gateway exposes clean REST endpoints for clients to manage templates, trigger notifications, fetch statuses, and update user preferences.

1. Create Notification Template

• Endpoint: POST /api/v1/templates
• Payload:
  JSON

  Copy

  {
    "clientId": "flipkart_app",
    "name": "iphone_sale_alert",
    "type": "PROMOTIONAL",
    "channel": "EMAIL",
    "content": "Hi {userName}, the new {phoneModel} is on sale!",
    "variables": ["userName", "phoneModel"]
  }
  

2. Send Notification Request

• Endpoint: POST /api/v1/notifications
• Payload:
  JSON

  Copy

  {
    "clientId": "flipkart_app",
    "templateId": "tmpl_98234",
    "recipientId": "user_109283",
    "channel": "PUSH",
    "priority": "HIGH",
    "variables": {
      "userName": "Alice",
      "phoneModel": "iPhone 15 Pro"
    },
    "scheduleTimestamp": null
  }
  

3. Query Notification Status

• Endpoint: GET /api/v1/notifications/{notificationId}
• Response: Returns current delivery status (PENDING, SENT, DELIVERED, FAILED).

4. Update User Notification Preferences

• Endpoint: PUT /api/v1/preferences
• Payload:
  JSON

  Copy

  {
    "clientId": "flipkart_app",
    "externalUserId": "user_109283",
    "preferences": {
      "email": true,
      "sms": false,
      "push": true
    }
  }
  

6. High-Level Design (HLD) Overview

At a high-level, our distributed microservices architecture decouples inbound client ingestion from asynchronous outbound delivery.

7. Low-Level Design (LLD) & Deep Dive Architecture

A. API Gateway Responsibilities

The API Gateway handles critical edge concerns before traffic hits internal microservices:

• Authentication & Authorization: Validates API keys and OAuth tokens.
• Rate Limiting: Protects downstream services against malicious traffic bursts and prevents overwhelming external providers.
• Routing: Dispatches requests via round-robin load balancing.

B. Template Service & Database Schema

Saves and versions message layouts across channels (since an SMS must be concise, whereas an HTML email can be rich with graphics).

CREATE TABLE templates (
    template_id SERIAL PRIMARY KEY,
    client_id VARCHAR(64) NOT NULL,
    name VARCHAR(128) NOT NULL,
    type VARCHAR(32) NOT NULL, -- PROMOTIONAL, TRANSACTIONAL
    channel VARCHAR(32) NOT NULL, -- EMAIL, SMS, PUSH
    content TEXT NOT NULL,
    variables JSONB,
    version INT NOT NULL,
    is_active BOOLEAN DEFAULT TRUE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

C. User Preference Service, Kafka, & Caching

Because millions of users modify their channel preferences frequently, direct database writes would saturate our storage layer.

1. Kafka Topic (user-preference-events): Inbound preference changes are published to a Kafka topic.
2. User Preference Consumer: Consumes events asynchronously and persists them to PostgreSQL.
3. Redis Preference Cache: Provider services check a Redis caching layer before dispatching messages, avoiding expensive DB reads on every single notification.

CREATE TABLE user_preferences (
    id SERIAL PRIMARY KEY,
    client_id VARCHAR(64) NOT NULL,
    external_user_id VARCHAR(128) NOT NULL,
    preferences JSONB NOT NULL, -- e.g., {"email": true, "sms": false, "push": true}
    updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
    UNIQUE(client_id, external_user_id)
);

D. The Outbox Pattern & Guaranteed Durability (Solving the Kafka Loss Problem)

The Danger of Naive Queueing

If the Notification Service accepts a request, pushes it directly to Kafka, and immediately returns 200 OK (Acknowledged) to the client—and Kafka crashes before consumers read the message—the notification is lost forever.

The Outbox Solution

To guarantee zero message loss, we implement the Transactional Outbox Pattern:

1. The Notification Service writes the request to two tables within a single database transaction: the notifications table and the outbox table.
2. Once committed, an acknowledgment is safely sent back to the client.
3. A CDC (Change Data Capture) pipeline (such as Debezium) streams new rows from the outbox table directly into our Kafka broker.

CREATE TABLE outbox (
    outbox_id SERIAL PRIMARY KEY,
    notification_id VARCHAR(64) NOT NULL,
    event_type VARCHAR(64) NOT NULL,
    payload JSONB NOT NULL,
    published BOOLEAN DEFAULT FALSE,
    created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP
);

E. Kafka Topics, Priority Segregation, & Routing

To guarantee that high-priority OTPs bypass promotional queues, we segregate messages using dedicated Kafka topics categorized by Severity and Channel.

Complete Topic Matrix

• Retry & DLQ (Dead Letter Queue): If a message consumption fails, it is routed to a retry queue with exponential backoff. After max retry exhaustion, it drops into a DLQ for manual engineering inspection.

F. External Service Integrations

Downstream provider workers talk to specialized industry gateways:

• SMS Providers: Twilio, MSG91
• Email Providers: SendGrid, Amazon SES, SMTP Servers
• Push Notification Providers:
  • FCM (Firebase Cloud Messaging) for Android devices.
  • APNS (Apple Push Notification Service) for iOS devices.

G. Delivery Status Tracking & Webhooks

To update notification statuses from SENT to DELIVERED, external providers use Webhooks.

1. External gateway sends an HTTP callback to our webhook endpoint (POST /api/v1/webhooks/delivery).
2. The delivery status is published to a Kafka delivery-status topic.
3. A consumer updates the notifications table (Final Status) and ingests micro-events into BigQuery (notification_event_db) for analytics and client reporting dashboards.

8. Summary and Key Takeaways

• Centralized Microservice Architecture: Decouples application logic from third-party notification mechanics, eliminating code duplication and vendor lock-in.
• CAP Theorem Trade-Off: We explicitly prioritize Availability (A) over strong consistency; minor replication delays in templates or user preferences do not compromise core system ingestion.
• Transactional Outbox Pattern: Solves the Kafka data-loss vulnerability by writing requests to a local DB outbox table before streaming via CDC.
• Priority Segregation: Separates critical bank OTP traffic from bulk promotional traffic using dedicated Kafka topics and isolated consumer pools.
• User Preference Enforcement: Utilizes Redis caching and asynchronous Kafka streams to respect user opt-in/opt-out rules without slowing down high-throughput dispatch pipelines.
• Resilient Error Handling: Implements exponential backoff retry queues and Dead Letter Queues (DLQs) to capture and diagnose poisoned or failed messages.