System Design Interview Prep for Beginners: Lab: Build a Notification System

This practical lab focuses on implementing a resilient, distributed Notification System. The architecture leverages an event-driven fan-out pattern to asynchronously dispatch alerts across multiple communication channels (Push and Email) using dedicated message queues, while introducing fault-tolerance mechanisms to handle message duplication and delivery failures via a Dead Letter Queue (DLQ).

1. Why Do We Need Distributed Notification Architecture?

In traditional monolithic applications, notification dispatching is often handled synchronously within the primary request-response lifecycle. For instance, when a user registers or makes a purchase, the main API server directly invokes an HTTP request to an external SMS, Push, or Email service provider (e.g., SendGrid, Firebase Cloud Messaging, Twilio).

Key Limitations of the Synchronous Approach

1. High Latency Overhead: Calling third-party communication APIs synchronously adds hundreds of milliseconds (or seconds) to the user's HTTP request.
2. Tight Coupling & Cascading Failures: If the email vendor experiences an outage or network throttling, the primary application server blocks, potentially depleting thread pools and crashing the core business service.
3. Lack of Resilience & Retries: If the notification network call fails mid-flight, retrying it inside the synchronous HTTP handler risks timing out the client or losing the notification permanently.

The Asynchronous Solution

By adopting an event-driven model, the API server simply publishes a lightweight domain event (e.g., OrderPlaced) to a message broker and returns an immediate response to the client. Dedicated downstream workers asynchronously pull tasks from isolated queues, decoupling payload ingestion from execution.

2. Core Mental Model: The Fan-Out Pattern

The Fan-Out Pattern is a messaging design pattern where a single incoming message or event is automatically duplicated and routed to multiple independent message queues. Each queue serves a specialized downstream worker or subsystem.

### Real-Life Analogy: The Central Press Engine
Imagine a central news agency that releases a major breaking news bulletin. Rather than having a single reporter manually call television networks, radio stations, and newspaper printing presses one by one, the agency posts the bulletin to a central distribution clearinghouse (the Exchange/Topic). 

From there, dedicated distribution channels instantly grab a copy:
- The TV station formats it for broadcast graphics.
- The Radio team scripts it for voice-over readouts.
- The Newspaper team formats it for print layouts.

Each channel operates at its own pace and uses its own specialized equipment. If the printing press jams (Email worker failure), it does not prevent the TV news from going live (Push worker success).

3. System Architecture & Topology Flow

The architecture decouples event generation from delivery using a message broker (such as RabbitMQ, Apache Kafka, or AWS SNS/SQS) with dedicated queue channels and a failure-isolation route.

4. Channel Isolation: Push vs. Email Queues

To prevent slow or failing channels from impacting performant channels, every delivery mechanism gets its own dedicated queue and worker pool.

5. Handling Duplicate Notifications (Idempotency)

In distributed messaging, message brokers generally offer At-Least-Once Delivery guarantees to prevent message loss. Consequently, network glitches, worker crashes right before sending an acknowledgement (ACK), or broker retries can cause the same notification message to be delivered to a worker twice.

The Risk of Non-Idempotent Processing

Without deduplication, a user might receive duplicate push notifications or duplicate receipt emails for a single transaction, leading to a poor user experience or duplicate billing warnings.

Implementing Idempotency via Deduplication Keys

To guarantee idempotent handling, workers check a fast, distributed key-value store (such as Redis) before sending out any notification.

Idempotent Worker Code Snippet

import redis
import json

# Initialize Redis client for deduplication caching
cache = redis.Redis(host='localhost', port=6379, db=0)

def process_notification_event(ch, method, properties, body):
    event = json.loads(body)
    notification_id = event.get("notification_id")
    user_id = event.get("user_id")
    message = event.get("message")
    
    # Construct an idempotency lock key with a 24-hour TTL
    dedup_key = f"dedup:notification:{notification_id}"
    
    # SETNX returns True if the key was set (i.e., not previously processed)
    is_first_time = cache.set(dedup_key, "PROCESSING", nx=True, ex=86400)
    
    if not is_first_time:
        print(f"[DUPLICATE DETECTED] Notification {notification_id} already processed. Skipping.")
        # Acknowledge broker to remove duplicate from queue
        ch.basic_ack(delivery_tag=method.delivery_tag)
        return

    try:
        # Execute actual delivery logic (Push or Email gateway call)
        send_external_notification(user_id, message)
        
        # Mark state as COMPLETED in cache
        cache.set(dedup_key, "COMPLETED", ex=86400)
        
        # Acknowledge successful processing
        ch.basic_ack(delivery_tag=method.delivery_tag)
        print(f"[SUCCESS] Notification {notification_id} sent successfully.")

    except Exception as exc:
        print(f"[ERROR] Delivery failed for {notification_id}: {exc}")
        # Remove lock key on transient failure so retry attempt can re-evaluate
        cache.delete(dedup_key)
        # Reject message; trigger broker retry/backoff mechanism
        ch.basic_nack(delivery_tag=method.delivery_tag, requeue=False)

def send_external_notification(user_id, message):
    # Simulated vendor HTTP API call
    pass

6. Managing Undeliverable Messages: Dead Letter Queue (DLQ)

When an event cannot be processed after a specified number of retry attempts (e.g., due to corrupt payloads, invalid target addresses, or persistent third-party outages), it must not block the queue or enter an infinite retry loop.

What is a Dead Letter Queue?

A Dead Letter Queue (DLQ) is a specialized secondary message queue designed to capture messages that the system cannot successfully process.

Note: A message is typically moved to a DLQ under three primary conditions:

1. The message is explicitly rejected (NACK or REJECT) by the worker with requeue = False.
2. The message exceeds its defined Maximum Retry Limit (e.g., 3 retries).
3. The message expires due to Time-To-Live (TTL) expiration in the main queue.

Purpose of the DLQ

1. Unblocks Main Pipelines: Prevents "poison pill" messages (malformed items that repeatedly crash workers) from consuming queue resources indefinitely.
2. Preserves Data: Ensures zero message loss; failed notifications remain securely stored for root-cause analysis.
3. Facilitates Replay Capabilities: Once downstream bug fixes or vendor outages resolve, operations teams can replay messages from the DLQ back into the primary fan-out exchange.

7. Step-by-Step Lab Execution Walkthrough

In this lab session, you will trigger the fan-out flow and observe real-time system responses under normal, duplicate, and fault conditions.

Step 1: Event Generation

An event payload is published to the central notification exchange:

{
  "notification_id": "evt_9876543210",
  "event_type": "ORDER_DISPATCHED",
  "user_id": "usr_4040",
  "email": "alex@example.com",
  "device_token": "fcm_token_abc123xyz",
  "payload": {
    "title": "Order Shipped!",
    "body": "Your item is on its way via Express Delivery."
  }
}

Step 2: Fan-Out Routing

The exchange duplicates the event payload into:

• Queue 1: notifications.push.queue
• Queue 2: notifications.email.queue

Step 3: Concurrent Worker Processing

• Push Worker picks up the event from notifications.push.queue, connects to FCM/APNs, and delivers a push alert to fcm_token_abc123xyz.
• Email Worker picks up the identical event from notifications.email.queue, injects variables into an HTML template, and sends an email via the SendGrid API.

Step 4: Duplicate Injection

The test script re-publishes the event with notification_id: "evt_9876543210".

• Both workers read the duplicate ID from Redis cache.
• The processing is safely bypassed without sending duplicate emails or push alerts to the end user.

Step 5: Failure & DLQ Simulation

A malformed device token event is injected into the system.

• The Push Worker attempts delivery, receives an unrecoverable HTTP 400 (Bad Device Token) from the push gateway, and exhausts maximum retry attempts.
• The message is automatically transferred to notifications.push.dlq.
• An operational alert is triggered, allowing engineers to inspect the payload in the DLQ UI dashboard.

8. Summary and Key Takeaways

Summary and Key Takeaways

• Fan-Out Architecture Decouples Systems: Publishing domain events to a centralized topic exchange allows independent worker channels (Push, Email, SMS) to scale and process messages asynchronously without blocking core business services.
• Isolated Queues Ensure Fault Containment: Allocating separate queues and worker pools for each notification vector prevents downstream delays or outages in one service (e.g., Email gateway latency) from degrading another (e.g., Push notifications).
• At-Least-Once Delivery Guarantees Duplicates: Distributed message brokers inherently risk duplicate deliveries; messaging components must be designed with explicit deduplication logic.
• Idempotency is Critical: Employing distinct idempotency keys stored in an in-memory cache (e.g., Redis) prevents sending redundant notifications to users when retries occur.
• DLQs Guarantee Zero Data Loss: Dead Letter Queues act as safety nets that store undeliverable or "poison pill" messages for inspection and manual/automated replay, keeping main processing channels clear.
• Exponential Backoff and Retries: Failure scenarios should incorporate bounded retry limits and exponential backoff to avoid hammering failing external vendor endpoints.
• Monitoring & Observability: Production systems require active alerting on DLQ volume increases and queue backlog depth to identify vendor degradation or payload schema bugs early.