Chapter 9: MESSAGE QUEUES & ASYNC WORKFLOWS - System Design Course: Complete Production Architecture | PlayBook Library

SLIDE 102: Message Queues: Why Use Them

Decoupling Producers and Consumers

Without Queue:

Structure & Concept Flow

graph LR

A[API] -->|Sync| B[Email Service]

B -->|Slow: 2 seconds| A

User waits 2 seconds for email to send.

With Queue:

Structure & Concept Flow

graph LR

A[API] -->|Enqueue: 5ms| B[Queue]

B --> C[Email Worker]

C --> D[Email Sent]

User gets response in 5ms. Email sends in background.

Queue Benefits:

Benefit	How
Decoupling	Producer and consumer independent
Buffering	Absorb traffic spikes
Reliability	Messages persist until processed
Scaling	Add more consumers for throughput

SLIDE 103: Message Queues: Delivery Guarantees

At-Least-Once vs Exactly-Once vs At-Most-Once

At-Most-Once (Fire and Forget):

Structure & Concept Flow

Send message -> Do not wait for ACK

Message may be lost if consumer crashes

Use for: Metrics, logs, non-critical notifications

At-Least-Once (Default for Most Queues):

Structure & Concept Flow

Send message -> Wait for ACK

If no ACK, redeliver message

Message may be processed multiple times

Use for: Most business operations (with idempotency)

Exactly-Once (Rare and Expensive):

Structure & Concept Flow

Send message -> Deduplicate -> Process -> ACK

Requires distributed transaction

Use for: Financial transactions (complex, slow)

SLIDE 104: Message Queues: Delivery Guarantees (continued)

Production Reality: At-Least-Once + Idempotency

CODE

def process_payment(message):
    payment_id = message["payment_id"]

    # Check if already processed (idempotency key)
    if db.exists("processed_payments", payment_id):
        return  # Already done, skip

    # Process payment
    charge_card(message["amount"])

    # Mark as processed
    db.insert("processed_payments", payment_id)

    # ACK message
    queue.ack(message)

Idempotency Key Strategies:

Strategy	Implementation
Message ID	Use queue-provided message ID
Business ID	Use order_id, payment_id, etc.
Content hash	Hash of message contents

Key Insight:
Exactly-once is a myth in distributed systems. Design for at-least-once with idempotent consumers.

SLIDE 105: Message Queues: Ordering Guarantees

When Order Matters

Scenario: User Updates Profile

CODE

Message 1: Set name = "Alice"
Message 2: Set name = "Bob"

If processed out of order:
Final name = "Alice" (wrong!)

Queue Ordering Options:

Queue Type	Ordering	Throughput
Standard Queue	No guarantee	High
FIFO Queue	Strict ordering	Lower
Partitioned Queue	Per-partition ordering	High

Partitioned Queue Pattern:

CODE

# All messages for same user go to same partition
partition_key = user_id

queue.send(
    message={"user_id": 123, "name": "Bob"},
    partition_key=partition_key
)

Messages with same partition key processed in order.

SLIDE 106: Message Queues: Ordering Guarantees (continued)

When to Use Each:

Use Standard Queue	Use FIFO/Partitioned
Independent events	State changes
Notifications	User profile updates
Analytics events	Order status changes
Log processing	Financial transactions

Handling Out-of-Order Messages:

CODE

def process_with_version(message):
    user_id = message["user_id"]
    version = message["version"]

    current_version = db.query(
        "SELECT version FROM users WHERE id = ?", user_id
    )

    if version <= current_version:
        # Old message, skip
        return

    # Apply update
    db.update(
        "UPDATE users SET name = ?, version = ? WHERE id = ?",
        message["name"], version, user_id
    )

Version numbers allow safe out-of-order processing.

SLIDE 107: Message Queues: Backpressure

When Producers Outpace Consumers

The Problem:

CODE

Producer: 10,000 messages/second
Consumer: 5,000 messages/second
Queue grows: 5,000 messages/second

After 1 hour: 18 million messages queued
Queue runs out of memory

Detecting Backpressure:

Metric	Healthy	Warning	Critical
Queue depth	< 1,000	1,000-10,000	> 10,000
Message age	< 1 min	1-5 min	> 5 min
Consumer lag	< 100	100-1,000	> 1,000

SLIDE 108: Message Queues: Backpressure Solutions

Solution 1: Scale Consumers

Structure & Concept Flow

graph LR

A[Queue: 10K msgs] --> B[Consumer 1]

A --> C[Consumer 2]

A --> D[Consumer 3]

A --> E[Consumer 4]

Add more consumer instances until queue drains.

Solution 2: Rate Limit Producers

CODE

def publish_with_backpressure(message):
    queue_depth = queue.get_depth()

    if queue_depth > 10000:
        # Reject or delay message
        raise BackpressureError("Queue full, try later")

    queue.send(message)

Solution 3: Drop Low-Priority Messages

CODE

def publish_with_priority(message, priority):
    queue_depth = queue.get_depth()

    if queue_depth > 5000 and priority == "low":
        # Drop low-priority messages
        return

    queue.send(message)

SLIDE 109: Background Workers: Job Processing Patterns

How Workers Process Messages

Basic Worker Loop:

CODE

class Worker:
    def run(self):
        while True:
            # 1. Fetch message from queue
            message = queue.receive(wait_time=20)

            if not message:
                continue

            try:
                # 2. Process message
                self.process(message)

                # 3. Acknowledge success
                queue.ack(message)

            except Exception as e:
                # 4. Handle failure
                self.handle_error(message, e)

    def process(self, message):
        # Business logic here
        pass

SLIDE 110: Background Workers: Job Processing Patterns (continued)

Worker Pool Pattern:

CODE

from concurrent.futures import ThreadPoolExecutor

class WorkerPool:
    def __init__(self, num_workers=10):
        self.executor = ThreadPoolExecutor(max_workers=num_workers)

    def run(self):
        while True:
            messages = queue.receive_batch(max_messages=10)

            futures = []
            for message in messages:
                future = self.executor.submit(self.process, message)
                futures.append((message, future))

            for message, future in futures:
                try:
                    future.result()
                    queue.ack(message)
                except Exception as e:
                    self.handle_error(message, e)

Process multiple messages concurrently within one worker.

Scaling Workers:

Metric	Action
Queue depth increasing	Add more workers
Workers idle	Remove workers
CPU/memory high	Larger instances

SLIDE 111: Background Workers: Retry Strategies

What to Do When Processing Fails

Retry with Exponential Backoff:

CODE

def process_with_retry(message):
    attempt = message.get("attempt", 1)
    max_attempts = 5

    try:
        do_work(message)
        queue.ack(message)

    except RetryableError as e:
        if attempt >= max_attempts:
            # Give up, send to DLQ
            dlq.send(message)
            queue.ack(message)
        else:
            # Retry with backoff
            delay = min(2 ** attempt, 300)  # Max 5 minutes
            message["attempt"] = attempt + 1
            queue.send(message, delay=delay)
            queue.ack(message)

    except FatalError as e:
        # Do not retry, send to DLQ
        dlq.send(message)
        queue.ack(message)

SLIDE 112: Background Workers: Retry Strategies (continued)

Backoff with Jitter:

CODE

def get_retry_delay(attempt):
    # Exponential backoff
    base_delay = 2 ** attempt

    # Cap at 5 minutes
    capped_delay = min(base_delay, 300)

    # Add jitter (0-50% of delay)
    jitter = random.uniform(0, capped_delay * 0.5)

    return capped_delay + jitter

Why Jitter:

CODE

Without jitter:
- 1000 messages fail at t=0
- All retry at t=2
- All fail again
- All retry at t=4 (thundering herd)

With jitter:
- 1000 messages fail at t=0
- Retry spread between t=2 and t=3
- Load distributed

Retry Delay Sequence:

Attempt	Base Delay	With Jitter
1	2 seconds	2-3 seconds
2	4 seconds	4-6 seconds
3	8 seconds	8-12 seconds
4	16 seconds	16-24 seconds
5	32 seconds	32-48 seconds

SLIDE 113: Background Workers: Idempotency

The Most Important Property

Why Idempotency Matters:

CODE

Message: "Charge customer $100"

Without idempotency:
- Message processed
- Worker crashes before ACK
- Message redelivered
- Customer charged $200 (disaster!)

With idempotency:
- Message processed
- Worker crashes before ACK
- Message redelivered
- Check: Already charged? Yes
- Skip, customer charged $100 (correct)

SLIDE 114: Background Workers: Idempotency (continued)

Idempotency Implementation Patterns:

Pattern 1: Idempotency Key in Database

CODE

def charge_customer(message):
    idempotency_key = message["idempotency_key"]

    # Check if already processed
    existing = db.query(
        "SELECT * FROM payments WHERE idempotency_key = ?",
        idempotency_key
    )

    if existing:
        return existing  # Already done

    # Process payment
    result = payment_provider.charge(message["amount"])

    # Store with idempotency key
    db.insert("payments", {
        "idempotency_key": idempotency_key,
        "result": result
    })

    return result

SLIDE 115: Background Workers: Idempotency (continued)

Pattern 2: Conditional Update

CODE

def update_inventory(message):
    product_id = message["product_id"]
    expected_quantity = message["expected_quantity"]
    new_quantity = message["new_quantity"]

    # Only update if current value matches expected
    result = db.execute("""
        UPDATE inventory
        SET quantity = ?
        WHERE product_id = ? AND quantity = ?
    """, new_quantity, product_id, expected_quantity)

    if result.rowcount == 0:
        # Already updated or different value
        return  # Idempotent: no change

Pattern 3: Natural Idempotency

CODE

def set_user_email(message):
    # Setting email to same value is naturally idempotent
    db.execute(
        "UPDATE users SET email = ? WHERE id = ?",
        message["email"], message["user_id"]
    )

Some operations are inherently idempotent. Identify and prefer these.

SLIDE 116: Background Workers: Poison Messages

Messages That Cannot Be Processed

What is a Poison Message:

CODE

A message that causes the consumer to fail every time it is processed.

Examples:
- Invalid JSON format
- Missing required field
- References deleted data
- Triggers application bug

The Danger:

CODE

1. Worker receives poison message
2. Worker crashes
3. Message returns to queue
4. Worker receives same message
5. Worker crashes again
6. Infinite loop, queue blocked

SLIDE 117: Background Workers: Poison Messages (continued)

Detecting Poison Messages:

CODE

def process_with_poison_detection(message):
    message_id = message["id"]

    # Track delivery count
    delivery_count = cache.incr(f"delivery:{message_id}")

    if delivery_count > 5:
        # This message has failed too many times
        dlq.send(message)
        queue.ack(message)
        alert("Poison message detected", message)
        return

    try:
        do_work(message)
        queue.ack(message)
        cache.delete(f"delivery:{message_id}")
    except Exception:
        # Will be redelivered
        raise

Queue Built-in Support:

Queue	Feature	Configuration
AWS SQS	maxReceiveCount	Move to DLQ after N failures
RabbitMQ	x-delivery-count	Track redeliveries
Kafka	Consumer tracking	Application-level

SLIDE 118: Dead Letter Queues: Purpose

Where Failed Messages Go

DLQ Flow:

Structure & Concept Flow

graph LR

A[Main Queue] --> B[Worker]

B -->|Success| C[Done]

B -->|Failure x5| D[Dead Letter Queue]

D --> E[Manual Review]

E --> F[Fix and Replay]

Why DLQ:

Purpose	How DLQ Helps
Preserve failed messages	Do not lose data
Unblock main queue	Poison messages removed
Enable investigation	Examine what failed
Allow replay	Reprocess after fix

SLIDE 119: Dead Letter Queues: Configuration

Setting Up DLQ

AWS SQS DLQ Configuration:

CODE

{
  "RedrivePolicy": {
    "deadLetterTargetArn": "arn:aws:sqs:us-east-1:123456789:my-dlq",
    "maxReceiveCount": 5
  }
}

After 5 failed attempts, message moves to DLQ.

DLQ Message Metadata:

CODE

{
  "body": "{original message}",
  "attributes": {
    "ApproximateReceiveCount": "5",
    "SentTimestamp": "1609459200000",
    "DeadLetterQueueSourceArn": "arn:aws:sqs:...:my-queue"
  }
}

Metadata helps diagnose why message failed.

SLIDE 120: Dead Letter Queues: Monitoring and Replay

DLQ Operations

Monitoring:

CODE

def monitor_dlq():
    depth = dlq.get_depth()

    if depth > 0:
        alert(f"DLQ has{depth} messages")

    if depth > 100:
        page_oncall(f"DLQ critical:{depth} messages")

Any messages in DLQ is a problem. Alert immediately.

Replay Strategy:

CODE

def replay_dlq():
    while True:
        message = dlq.receive()
        if not message:
            break

        # Optionally transform message
        fixed_message = fix_message(message)

        # Send back to main queue
        main_queue.send(fixed_message)

        # Remove from DLQ
        dlq.ack(message)

SLIDE 121: Dead Letter Queues: Monitoring and Replay (continued)

Replay Considerations:

Consideration	Action
Bug in consumer	Fix bug before replay
External dependency down	Wait for recovery
Invalid data	Transform or discard
Ordering matters	Replay in order

DLQ Retention:

CODE

- Keep messages in DLQ for 14 days (investigation time)
- Archive to S3 before deletion (compliance)
- Document why messages failed

Key Insight:
DLQ is not a trash can. It is a hospital for sick messages. Investigate every message, fix root cause, replay or discard intentionally.

SLIDE 122: Eventual Consistency: User Experience

How Users Experience Async Systems

The Expectation vs Reality:

User Expects	Reality
“I posted, I should see it”	Post may not appear for 1-5 seconds
“I liked, count should increase”	Count updates eventually
“I sent money, balance should change”	Balance updates after confirmation

Making Eventual Consistency Invisible:

Pattern 1: Optimistic UI

CODE

function likePost(postId) {
  // Update UI immediately (optimistic)
  setLikeCount(likeCount + 1);
  setLiked(true);

  // Send to server in background
  api.post(`/posts/${postId}/like`).catch(() => {
    // Rollback on failure
    setLikeCount(likeCount);
    setLiked(false);
  });
}

SLIDE 123: Eventual Consistency: User Experience (continued)

Pattern 2: Read Your Writes

CODE

def create_post(user_id, content):
    post = db.insert("INSERT INTO posts ...")

    # Store in user's local view
    cache.lpush(f"user_posts:{user_id}", post.id)

    return post

def get_user_posts(user_id):
    # Include locally created posts
    local_posts = cache.lrange(f"user_posts:{user_id}", 0, -1)
    db_posts = db.query("SELECT * FROM posts WHERE user_id = ?", user_id)

    # Merge and deduplicate
    return merge(local_posts, db_posts)

User always sees their own posts, even before replication completes.

Pattern 3: Explicit Loading States

CODE

function Post({ postId }) {
  const { data, isLoading } = usePost(postId);

  if (isLoading) {
    return <Skeleton />; // Show loading state
  }

  return <PostContent data={data} />;
}

Tell user something is happening, do not show stale data as if fresh.

SLIDE 124: Eventual Consistency: When It Breaks

Scenarios Where Eventual Consistency Fails

Scenario 1: Inventory Oversell

CODE

Item quantity: 1
User A reads: 1 available
User B reads: 1 available (stale)
User A purchases: quantity = 0
User B purchases: quantity = -1 (oversold!)

Solution: Strong consistency for inventory with pessimistic locking.

Scenario 2: Double Spend

CODE

Balance: $100
User reads balance: $100
User initiates transfer A: $80
User initiates transfer B: $50 (before A completes)
Both succeed: Balance = -$30

Solution: Synchronous balance check with transaction.

SLIDE 125: Eventual Consistency: When It Breaks (continued)

Scenario 3: Duplicate Actions

CODE

User clicks "Submit Order" twice quickly
Both requests processed
Two orders created

Solution: Idempotency key on client.

CODE

function submitOrder() {
  const idempotencyKey = generateUUID();

  api.post("/orders", {
    items: cart,
    idempotency_key: idempotencyKey,
  });
}

When to Use Strong Consistency:

Use Case	Why
Financial transactions	Money must not be lost or duplicated
Inventory management	Cannot sell what you do not have
Booking systems	Double booking ruins trust
User authentication	Security critical

SLIDE 126: Caching and Async: Summary

Key Takeaways

Caching:

Use cache-aside as default pattern
TTL is a trade-off between freshness and load
Invalidation is hard: prefer TTL + explicit delete
Handle thundering herd with jitter, locks, or stale-while-revalidate
Warm cache before expected traffic

CDN:

Cache static assets with long TTL and versioned URLs
Use Cache-Control headers to control behavior
Purge only for emergencies, not routine updates

SLIDE 127: Caching and Async: Summary (continued)

Message Queues:

Design for at-least-once with idempotent consumers
Use partitioned queues when order matters
Monitor queue depth for backpressure
Scale consumers based on queue depth

Background Workers:

Implement exponential backoff with jitter
Idempotency is mandatory, not optional
Detect and handle poison messages
Use dead letter queues for failed messages

Eventual Consistency:

Use optimistic UI for better user experience
Implement read-your-writes for user’s own data
Use strong consistency for financial/inventory operations
Design for acceptable staleness, not zero staleness

Coming Up Next: Section 5 - Reliability and Failure Handling