System Design Interview Prep: Interview Questions in a Notification System

As a system design interview reaches its final stages, interviewers frequently introduce probing follow-up questions. These scenarios test a candidate's ability to critically evaluate architectural bottlenecks, handle external provider failures, and address fundamental distributed systems limitations like "exactly-once" delivery constraints.

1. The Problem of Mixed Notification Queues

Status Quo and the Priority Dilemma

During the initial phases of system design, engineers often group notifications into a single, unified processing queue per channel. However, real-world notifications possess radically different urgency and patience levels.

When a marketing campaign drops 1 million promotional notifications into the queue simultaneously, time-sensitive alerts—such as an urgent user login authentication code requested by a user named Allan—end up sitting directly behind a massive backlog of marketing notifications.

Real-Life Analogy: Emergency vs. Regular Mail

Imagine a post office handling both emergency medical supplies and bulk catalog advertisements. If both are loaded into the same delivery truck in the exact order they arrived, life-saving medicine gets delayed behind thousands of junk mail flyers. To solve this, sorting facilities use separate priority lanes.

2. Decoupling Traffic: Separate Queues and Dedicated Workers

To resolve the queue congestion problem, the system architecture must be modified to separate operational traffic streams completely.

Architectural Evolution

Instead of maintaining a single queue per channel (e.g., SMS, Email, Push), the system must split each channel into two independent queues:

1. Transactional Queue: Reserved exclusively for high-priority, time-sensitive events (OTP codes, account alerts, security warnings).
2. Marketing Queue: Dedicated to promotional campaigns, newsletters, and non-urgent announcements.

By assigning dedicated workers specifically to the transactional queue—workers that marketing jobs can never borrow or starve—the system guarantees that an influx of 1 million marketing emails will have zero impact on the delivery latency of login verification codes.

Note: This pattern mirrors production-grade architectures such as Uber's priority buckets.

3. External Provider Outages: The 1-Hour Failure Scenario

Suppose an external email service provider goes completely offline for 1 hour. Interviewers often ask: "Walk me through precisely what your system does during that hour."

Key System Behaviors During an Outage

• Durability Prevents Data Loss: Because the underlying queue is durable (backed by persistent storage like disk-backed message brokers), notifications do not disappear; they accumulate safely in the queue.
• Worker Retry Storms: Without proper safeguards, hundreds of background workers will repeatedly hammer the dead provider's API endpoint for 60 straight minutes with zero successful outcomes, burning CPU cycles and clogging internal network resources.

4. Introducing Circuit Breakers and Health Probes

To prevent worker retry storms during extended provider outages, resilient systems incorporate the Circuit Breaker design pattern.

How Circuit Breakers Function

1. Failure Threshold: After a configured number of consecutive connection failures, the circuit breaker trips and "opens," causing workers to immediately fail fast without calling the dead provider.
2. Single Probe Recovery: While the circuit is open, a single lightweight probe call is sent out every minute (or defined interval) to check if the provider has come back online. Once the probe succeeds, the circuit closes, and normal traffic resumes.

5. Multi-Provider Redundancy Strategies

Relying on a single third-party notification vendor creates a single point of failure. Enterprise-grade architectures implement redundancy strategies to mitigate provider downtime:

Note: A login code valid for 60 seconds is completely useless if delivered an hour late due to an outage. Workers must check message timestamps and drop expired notifications instantly upon dequeue.

6. The Myth of "Exactly-Once" Delivery and Idempotency Failures

Interviewers frequently challenge candidates on distributed systems guarantees: "You keep saying exactly-once delivery is impossible. Show me the exact case where your delivery fails and your idempotency key breaks down."

The Failure Walkthrough

Consider this step-by-step race condition with an SMS provider:

1. Step 1: A worker picks up a ticket and checks Redis. It finds that this notification ID has not yet been sent.
2. Step 2: The worker writes to Redis, marking the notification as "picked up".
3. Step 3: The worker calls the SMS provider API and hits a network timeout before receiving a response.

The Dilemma

Did the SMS actually reach the user's phone, or did it fail before transmission? We cannot know with 100% certainty.

• If our system retries, Allan might receive duplicate SMS messages.
• If our system does not retry, Allan might receive nothing at all.

7. Mitigating Duplicates: Idempotency Keys and Carrier Realities

While distributed systems cannot guarantee true exactly-once delivery across unreliable networks, engineers layer multiple defenses to achieve at-least-once delivery with deduplication safeguards.

Defense Layers Against Duplicates

• Provider-Level Idempotency Keys: Modern email and SMS APIs accept a unique client-provided idempotency key (such as our unique notification_id). If the provider receives the same ID twice, it drops the duplicate request instead of resending the message.
• Telecom Network Anomalies: Even if our backend and our API vendor prevent duplicates, downstream telecom carriers (like AT&T, Verizon, or international cellular networks) can occasionally duplicate SMS delivery on congested or faulty network days.
• Product Requirement Framing: This inherent hardware/network unpredictability is why production requirements specify that users should ideally never see the same notification twice—acknowledging that edge cases in distributed infrastructure can occasionally bypass software-level protections.

Summary and Key Takeaways

• Separate Workloads: Always assign separate queues and dedicated workers for transactional versus marketing notifications to prevent bulk campaigns from starving urgent alerts.
• Durability Protects Data: When an external provider goes down, durable queues safely hold accumulated messages so nothing is lost.
• Circuit Breakers Stop Retry Storms: Use circuit breakers to stop workers from hammering dead endpoints, paired with periodic probe calls to detect recovery.
• Multi-Provider Redundancy: Enterprise systems maintain secondary standby providers or active-active routing setups to ensure high availability.
• Respect Message TTLs: Time-sensitive messages (like OTPs) must carry expiry timestamps so workers can drop stale payloads rather than delivering useless messages.
• Acknowledge Network Limits: True exactly-once delivery is physically impossible in distributed network topologies due to unacknowledged timeouts.
• Leverage API Idempotency: Pass unique notification IDs as idempotency keys to external providers to minimize duplicate dispatches.
• Design for "Ideally": Accept that carrier-level anomalies (telecom network glitches) mean duplicate prevention is a best-effort architectural goal rather than an absolute guarantee.