DISTRIBUTED TRANSACTIONS & SAGAS

SLIDE 85: Section Intro

Section 6: Distributed Transactions, Sagas, and Consistency

Section 5 established that each service owns its own database. This section confronts the hardest consequence of that rule: when a business operation spans multiple services, how do you keep data consistent across them without a shared transaction?

In a monolith, you wrap everything in a database transaction. If payment fails, the order is rolled back, inventory is released, and the database is consistent. In microservices, there is no shared transaction. Payment lives in one database, inventory in another, orders in a third. When payment fails after inventory has already been reserved, you need a strategy that goes far beyond BEGIN and ROLLBACK.

This section teaches that strategy. It is the highest-signal material in this course for senior-level interviews.

What We Will Cover:

1. Why local ACID transactions do not work across service boundaries
2. Two-phase commit (2PC) and why it is usually avoided
3. The saga pattern: orchestration and choreography
4. Compensating transactions for undoing completed steps
5. The transactional outbox and inbox patterns for reliable messaging
6. Idempotency, at-least-once delivery, and the exactly-once myth
7. Dead-letter queues, retry strategies, and partial failure handling
8. State machines for managing service lifecycles
9. A complete worked example: e-commerce checkout saga with failure cases
10. Human intervention for unrecoverable failures

Where This Appears in Real Systems:

Every time you place an order on a platform like Amazon, a saga coordinates order creation, inventory reservation, payment authorization, and shipment scheduling across independent services. When payment fails, compensating transactions release inventory. When shipping fails after payment, the system must decide between retrying, refunding, or escalating to a human operator. This is the reality of distributed consistency.

SLIDE 86: Why Local ACID Is Not Enough

In a monolith, a single database transaction guarantees four properties (ACID): Atomicity (all or nothing), Consistency (valid state after every transaction), Isolation (concurrent transactions do not interfere), and Durability (committed data survives crashes).

The monolith checkout in one transaction:

BEGIN TRANSACTION;
  INSERT INTO orders (order_id, customer_id, total) VALUES ('ord_123', 'cust_456', 59.98);
  UPDATE inventory SET reserved = reserved + 1 WHERE product_id = 'prod_789';
  INSERT INTO payments (payment_id, order_id, amount, status) VALUES ('pay_001', 'ord_123', 59.98, 'AUTHORIZED');
COMMIT;

If any statement fails, the entire transaction rolls back. The database is never in a state where an order exists without a payment, or where inventory is reserved without an order.

Why this breaks in microservices:

The orders table is in the Order Service database. The inventory table is in the Inventory Service database. The payments table is in the Payment Service database. There is no single BEGIN TRANSACTION that spans three separate databases on three separate servers.

You cannot roll back a payment that was authorized on Stripe’s servers by rolling back your local database transaction. You cannot release inventory in another service’s database by issuing a ROLLBACK in yours.

The core challenge: In microservices, each service can guarantee ACID within its own database. But no mechanism guarantees ACID across multiple services’ databases. Everything in this section exists to handle that gap.

Interview Relevance: When you propose writing data across multiple services in an interview, the interviewer expects you to immediately address consistency. If you do not, they will ask, and it signals you have not thought about the hardest part of the design.

SLIDE 87: Two-Phase Commit (2PC)

Two-phase commit is a protocol designed to achieve atomic transactions across multiple databases. It is important to understand it because interviewers ask about it, even though it is rarely used in modern microservices.

Phase 1 (Prepare): The coordinator asks each participant: “Can you commit this transaction?” Each participant acquires locks, validates the data, and responds “Ready” or “Abort.”

Phase 2 (Commit): If all participants said “Ready,” the coordinator sends “Commit” to all of them. If any participant said “Abort,” the coordinator sends “Rollback” to all of them.

What it guarantees: Either all participants commit or all participants roll back. The transaction is atomic across all databases.

Interview Relevance: Understanding 2PC is necessary because interviewers test whether you know why it exists and why it is avoided. The next slide explains why.

SLIDE 88: Why 2PC Is Usually Avoided

2PC guarantees atomicity, but the guarantees come at a cost that makes it impractical for most microservices architectures.

Problem 1: Blocking and locks

During Phase 1, each participant holds locks on the rows involved in the transaction. These locks are held until Phase 2 completes. If the coordinator crashes between Phase 1 and Phase 2, participants hold their locks indefinitely, blocking all other transactions on those rows. In a high-throughput system processing thousands of orders per second, this lock contention is a severe bottleneck.

Problem 2: Coordinator is a single point of failure

If the coordinator crashes after sending “Prepare” but before sending “Commit,” participants are stuck in an uncertain state. They have said “Ready” and are holding locks, but they do not know whether to commit or roll back. Recovery requires a new coordinator to read the transaction log and resume, which adds complexity and delay.

Problem 3: Latency

2PC requires two network round trips to every participant. In a microservices system where participants might be in different data centers, each round trip adds latency. A checkout that takes 50ms in a monolith might take 300-500ms with 2PC.

Problem 4: Reduced availability

If any single participant is unavailable, the entire transaction cannot proceed. With 4 participants each at 99.9% availability, the combined availability of the transaction is 99.6%. 2PC trades availability for consistency, which conflicts with the microservices goal of independent availability.

The practical reality: 2PC works for tightly controlled environments (two databases in the same data center, managed by the same team). It does not work well for independently deployed microservices with separate databases, separate teams, and separate availability requirements.

Interview Tip: In an interview, explain 2PC briefly, then explain why you would use sagas instead. “2PC guarantees atomicity but creates a blocking, latency-heavy, availability-reducing protocol that conflicts with microservices independence. Sagas achieve eventual consistency without distributed locks.” This shows depth.

Interview Relevance: The ability to explain why 2PC is avoided, not just that it is avoided, separates candidates who understand distributed systems from those who memorized a pattern name.

SLIDE 89: The Distributed Transaction Problem

Before learning the solution (sagas), you need to fully understand the problem. Here is a concrete scenario that illustrates why distributed consistency is hard.

Scenario: E-commerce checkout

A customer places an order. Three services must update their data:

1. Order Service: Create the order record (status: PENDING)
2. Inventory Service: Reserve 2 units of product XYZ
3. Payment Service: Authorize $59.98 on the customer’s credit card

The happy path works fine. All three services succeed. The order is confirmed.

Now consider the failure cases:

Case 1: Order created, inventory reserved, payment fails (insufficient funds). The order exists. Inventory is reserved. But there is no payment. The customer is not charged, but the product is held and unavailable to other customers.

Case 2: Order created, inventory reservation fails (out of stock). The order exists with no inventory backing it. If we proceed to payment, we charge the customer for something we cannot ship.

Case 3: Order created, inventory reserved, payment authorized, then the Order Service crashes before updating the order status to CONFIRMED. The payment was charged, inventory is reserved, but the order shows PENDING. The customer is charged but sees no confirmation.

Each failure case requires a different recovery action. There is no single “rollback” that fixes everything. You need compensating transactions: cancel the order, release the inventory, refund the payment. And each compensating action can itself fail.

This is the problem sagas solve. Not with a single distributed transaction, but with a sequence of local transactions and compensating actions.

Interview Relevance: Describing these specific failure cases in an interview before proposing your solution shows the interviewer you understand the problem deeply, not just the pattern name.

SLIDE 90: The Saga Pattern

A saga is a sequence of local transactions, where each transaction updates one service’s database, and each transaction has a corresponding compensating transaction that can undo its effect if a later step fails.

The key insight: A saga does not guarantee atomicity in the ACID sense. It guarantees that the system will eventually reach a consistent state, either by completing all steps (success) or by running compensating transactions for all completed steps (failure recovery).

How a saga differs from a transaction:

• A database transaction is all-or-nothing. Either everything commits or everything rolls back.
• A saga is a sequence of committed local transactions. Each step commits immediately in its own database. If step 3 fails, steps 1 and 2 have already committed. You cannot roll them back. You must compensate them.

Compensation is not the same as rollback:

• A database rollback undoes uncommitted changes as if they never happened.
• A compensating transaction creates a new transaction that semantically reverses the effect of a committed transaction.
• Cancelling an order is not rolling back the INSERT. It is creating a new state change: UPDATE order SET status = ‘CANCELLED’.
• Refunding a payment is not undoing the charge. It is creating a new refund transaction on the customer’s credit card.

The two implementation approaches:

1. Orchestration: A central coordinator service tells each participant what to do and handles failures. Covered in Slide 92.
2. Choreography: Each service reacts to events from the previous step. No central coordinator. Covered in Slide 93.

Interview Relevance: When you mention sagas in an interview, immediately clarify that a saga is not a distributed transaction with rollback. It is a sequence of committed local transactions with compensating actions for failure recovery. This distinction matters.

SLIDE 90B: Saga Pattern, Step-by-Step

How a saga executes:

Forward flow (success):

Each step executes and commits in its own service database. Step 1 commits in the Order DB. Step 2 commits in the Inventory DB. Step 3 commits in the Payment DB. Step 4 updates the Order DB to CONFIRMED.

Backward flow (failure at Step 3):

Payment authorization fails. Steps 1 and 2 have already committed (the order exists, inventory is reserved). The saga runs compensating transactions in reverse order: release the reserved inventory (compensating step 2), then cancel the order (compensating step 1).

What each step looks like:

• Step 1 (Create Order): Insert order with status PENDING. Compensating action: Update order status to CANCELLED.
• Step 2 (Reserve Inventory): Decrement available stock, increment reserved stock. Compensating action: Increment available stock, decrement reserved stock.
• Step 3 (Authorize Payment): Call payment gateway to authorize the charge. Compensating action: Call payment gateway to void the authorization.
• Step 4 (Confirm Order): Update order status to CONFIRMED. No compensating action needed (this is the final step).

Interview Relevance: Being able to list each saga step with its corresponding compensating action for a concrete scenario is the level of detail interviewers expect at the senior level.

SLIDE 90C: Interview Practice

Question:

An interviewer asks: “How is a saga different from a distributed transaction? And what happens if a compensating transaction itself fails?”

Take 60 seconds to think about both parts before reading below.

Weak Answer:

“A saga is basically a distributed transaction but using events instead of 2PC. If a compensating transaction fails, we retry it.”

Why this is weak: Saying a saga is “basically a distributed transaction” is incorrect. They have fundamentally different guarantees. And “retry it” does not address what happens if retries keep failing.

Strong Answer:

“A saga and a distributed transaction are fundamentally different. A distributed transaction (2PC) holds locks across all participants and guarantees atomicity, meaning either everything commits or nothing does. The data is never in an intermediate state visible to other transactions.

A saga commits each step immediately in its own database. After step 2 commits, that data is visible to the rest of the system. If step 3 fails, steps 1 and 2 are already committed and visible. You cannot roll them back. You must run compensating transactions to semantically reverse their effects. During the compensation window, the system is in a temporarily inconsistent state. This is the trade-off: sagas give up atomicity for availability and independence.

For the second part: if a compensating transaction fails, you retry it with exponential backoff. Compensating transactions must be designed to be idempotent, so retrying is always safe. If retries keep failing (the Inventory Service is down for an extended period), the failed compensation goes to a dead-letter queue. An alert fires. A human operator or an automated recovery process handles it when the downstream service recovers. In practice, this is rare, but the system must have a path for it. You never silently drop a failed compensation.”

What makes this strong: The candidate explains the fundamental difference (committed vs uncommitted intermediate states), acknowledges the temporary inconsistency trade-off, and addresses the failure-of-compensation scenario with a practical answer (retry, DLQ, human intervention).

SLIDE 91: Compensating Transactions

A compensating transaction is a new transaction that semantically reverses the effect of a previously committed transaction. It does not undo the original transaction. It creates a new action that counteracts it.

Why “undo” is the wrong mental model:

In a database, ROLLBACK discards uncommitted changes. The data reverts to its previous state as if the transaction never happened. Compensating transactions cannot do this because the original transaction has already been committed. Other services may have already read the committed data and acted on it.

Examples of compensating transactions:

• Original: Create order (status: PENDING). Compensation: Update order status to CANCELLED. The order record still exists in the database. It is not deleted.
• Original: Reserve 2 units of inventory. Compensation: Release 2 units back to available stock. The reservation record may be kept for auditing.
• Original: Authorize $59.98 payment. Compensation: Void the authorization (if not yet captured) or refund the charge (if already captured). The payment record shows both the charge and the reversal.
• Original: Send confirmation email. Compensation: There is none. You cannot unsend an email. This is why notification should be the last step in a saga, or sent only after the saga is fully confirmed.

Design rules for compensating transactions:

1. Every saga step must have a defined compensating action before you implement the saga. If you cannot define how to compensate a step, reconsider whether it belongs in the saga.
2. Compensating transactions must be idempotent. Running “release inventory for order ord_123” twice should produce the same result as running it once.
3. Some actions are not compensable. Sending an email, calling a third-party API with side effects, or shipping a physical product cannot be reversed. Design the saga so non-compensable steps happen last.

Interview Relevance: Explaining that compensation is not rollback, and listing which steps are compensable versus which are not, demonstrates a practical understanding of saga design that most candidates lack.

SLIDE 92: Orchestration

In the orchestration approach, a central coordinator (the orchestrator) manages the saga by telling each service what to do and handling the results.

How the orchestrator works:

The orchestrator holds the saga’s state machine. It knows which step to execute next, what to do if a step succeeds, and what compensating actions to trigger if a step fails. It sends commands to each service (“create this order,” “reserve this inventory”) and waits for responses.

Advantages of orchestration:

• The entire workflow is visible in one place (the orchestrator’s code)
• Easy to understand the flow by reading the orchestrator logic
• Easy to add new steps or change the order
• Centralized error handling and compensation logic

Disadvantages of orchestration:

• The orchestrator can become a single point of failure
• Risk of the orchestrator accumulating too much business logic (it should coordinate, not compute)
• Tighter coupling: the orchestrator must know about every service in the saga

When to use orchestration: Complex sagas with many steps (5+), conditional branching, or intricate failure handling. The visibility of a central coordinator outweighs the coupling cost.

Interview Relevance: When you describe a saga in an interview, the interviewer will ask whether you would use orchestration or choreography. Having a clear opinion with reasoning is expected.

SLIDE 93: Choreography

In the choreography approach, there is no central coordinator. Each service listens for events and reacts by executing its step and publishing the next event.

How choreography works:

No service tells another service what to do. Each service publishes an event describing what happened. Other services subscribe to events they care about and decide independently how to respond. The Order Service publishes “OrderCreated.” The Inventory Service hears it and reserves inventory. It publishes “InventoryReserved.” The Payment Service hears it and authorizes payment.

Advantages of choreography:

• No single point of failure (no orchestrator to crash)
• Loose coupling: services only know about events, not about each other
• Easy to add new consumers without changing existing services
• Each service is autonomous

Disadvantages of choreography:

• The overall workflow is spread across multiple services. No single place shows the full flow.
• Debugging is harder. You must trace events across services to understand what happened.
• Complex compensation logic is distributed. Each service must know when and how to compensate.
• Cyclic event chains can be hard to detect and prevent.

When to use choreography: Simple sagas with 2-3 steps where the flow is straightforward and failure handling is simple. The loose coupling and autonomy outweigh the reduced visibility.

Interview Relevance: Choosing between orchestration and choreography is a design decision that interviewers test. The next slide provides the comparison framework.

SLIDE 94: Orchestration vs Choreography

The practical reality:

Most production systems use orchestration for their most critical workflows (checkout, payment, fulfillment) and choreography for simpler, less critical flows (analytics updates, notification triggers, cache invalidation).

Many teams start with choreography because it feels simpler, then migrate to orchestration when the workflow grows to 5+ steps and debugging event chains becomes painful.

Interview Tip: A strong interview answer is: “I would use orchestration for the checkout saga because it has 5 steps with complex failure handling, and I need the full workflow visible in one place. For downstream processes like sending notifications and updating analytics, I would use choreography because those are simple, independent reactions to the OrderConfirmed event.”

Interview Relevance: Choosing the right approach for different workflows within the same system, rather than picking one approach for everything, shows mature architectural judgment.

SLIDE 95: The Transactional Outbox Pattern

The transactional outbox solves a critical reliability problem: how do you update your database and publish an event atomically? Without it, you can lose events or publish events for changes that were rolled back.

The problem:

1. Order Service inserts order into database
2. Order Service publishes OrderCreated event to message broker

What if the database insert succeeds (step 1) but the event publish fails (step 2)? The order exists but no other service knows about it. Inventory is not reserved. Payment is not authorized. The order is stuck.

What if the event publishes (step 2) but the database insert fails (step 1)? Other services start processing an order that does not exist.

The solution: Write the event to the database, not the message broker.

Instead of publishing directly to the message broker, the Order Service writes the event to an outbox table in the same database as the order, within the same transaction.

A separate process (the message relay or CDC connector) reads the outbox table and publishes events to the message broker. After successful publication, the relay marks the outbox entry as sent.

Why this works: The order insert and the outbox insert are in the same database transaction. They either both commit or both roll back. You never have an order without an outbox entry, and you never have an outbox entry without an order.

Interview Relevance: The transactional outbox is a specific pattern that interviewers expect senior candidates to know. It shows you understand that publishing an event and writing to a database are two separate operations that can fail independently.

SLIDE 95B: Transactional Outbox, Implementation

The outbox table:

CREATE TABLE outbox (
    event_id UUID PRIMARY KEY,
    aggregate_type VARCHAR(100),
    aggregate_id VARCHAR(100),
    event_type VARCHAR(100),
    payload JSONB,
    created_at TIMESTAMP DEFAULT NOW(),
    published BOOLEAN DEFAULT FALSE
);

Writing to the outbox within the same transaction as the business data:

def create_order(customer_id, items):
    order_id = generate_uuid()
    event_id = generate_uuid()

    db.begin_transaction()
    db.execute(
        "INSERT INTO orders (order_id, customer_id, status) VALUES (?, ?, ?)",
        order_id, customer_id, "PENDING"
    )
    db.execute(
        "INSERT INTO outbox (event_id, aggregate_type, aggregate_id, event_type, payload) VALUES (?, ?, ?, ?, ?)",
        event_id, "Order", order_id, "OrderCreated",
        json.dumps({"order_id": order_id, "customer_id": customer_id, "items": items})
    )
    db.commit_transaction()

The order insert and the outbox insert succeed or fail together. There is no window where one exists without the other.

The message relay:

A separate process polls the outbox table for unpublished events, publishes them to the message broker, and marks them as published. Alternatively, Change Data Capture (CDC) tools like Debezium can tail the database transaction log and publish outbox entries automatically without polling.

What about duplicate events?

The relay might publish an event and crash before marking it as published. On restart, it publishes the same event again. This means consumers must be idempotent, which is a requirement we will address in the next slides.

Interview Relevance: Showing the actual table schema and transaction code in an interview demonstrates implementation-level understanding, not just conceptual awareness.

SLIDE 95C: Interview Practice

Question:

An interviewer asks: “Your Order Service creates an order and needs to notify the Inventory Service to reserve stock. How do you guarantee that the event is always published when the order is created?”

Take 60 seconds to think about the reliability problem before reading below.

Weak Answer:

“After inserting the order into the database, I publish an event to Kafka. If the publish fails, I retry.”

Why this is weak: There is a fundamental gap between the database commit and the Kafka publish. If the application crashes after the database commit but before the Kafka publish, the event is lost permanently. Retrying only works if the application is still running. The answer does not address the atomicity problem.

Strong Answer:

“I would use the transactional outbox pattern. Instead of publishing directly to Kafka, I write the event to an outbox table in the same database as the order, within the same transaction. Both the order row and the outbox row either commit together or roll back together.

A separate message relay process reads the outbox table and publishes events to Kafka. After successful publication, it marks the outbox entry as sent. If the relay crashes before marking it, it republishes on restart, which means consumers might receive duplicate events. So consumers must be idempotent, processing the same event twice produces the same result as processing it once.

Alternatively, I could use Debezium with Change Data Capture to tail the PostgreSQL write-ahead log and publish outbox entries automatically. This avoids polling latency and is the approach many production systems use.”

What makes this strong: The candidate identifies the atomicity problem (database commit and event publish are separate operations), proposes a concrete solution (outbox), explains the duplicate event consequence, and mentions a production-grade implementation (Debezium CDC).

SLIDE 96: The Inbox Pattern and Message Deduplication

The outbox pattern guarantees that events are published at least once. But “at least once” means the same event might be delivered to a consumer more than once. The inbox pattern handles this on the consumer side.

Why duplicates happen:

• The message relay publishes an event to Kafka but crashes before marking it as sent. On restart, it publishes again.
• Kafka delivers a message to a consumer, but the consumer crashes before committing its offset. Kafka redelivers.
• Network issues cause a retry that results in the broker receiving the same message twice.

The inbox pattern:

Each consumer maintains an inbox table that records the ID of every event it has processed.

def handle_order_created(event):
    if db.exists("SELECT 1 FROM inbox WHERE event_id = ?", event.event_id):
        return

    db.begin_transaction()
    db.execute("INSERT INTO inbox (event_id, processed_at) VALUES (?, NOW())", event.event_id)
    db.execute(
        "UPDATE inventory SET reserved = reserved + ? WHERE product_id = ?",
        event.quantity, event.product_id
    )
    db.commit_transaction()

Before processing an event, the consumer checks if the event ID is already in the inbox. If it is, the event has already been processed and is skipped. If it is not, the consumer inserts the event ID into the inbox and processes the event in the same transaction.

Inbox table maintenance: Over time, the inbox table grows. Periodically purge entries older than the message broker’s retention period (e.g., if Kafka retains messages for 7 days, purge inbox entries older than 7 days).

Interview Relevance: Mentioning the inbox pattern alongside the outbox pattern shows the interviewer you understand the full reliability chain: reliable publishing (outbox) and reliable consumption (inbox).

SLIDE 97: Idempotency Keys

Idempotency is the property that executing an operation multiple times produces the same result as executing it once. In microservices with at-least-once delivery, idempotency is not optional. It is a requirement for correctness.

Operations that are naturally idempotent:

• Setting a value: “Set user name to Jonathan” produces the same result whether run once or five times.
• Deleting a record by ID: “Delete order ord_123” succeeds once and is a no-op afterward.
• Reading data: Always idempotent.

Operations that are not naturally idempotent:

• Incrementing a value: “Add $10 to account balance” gives a different result each time.
• Creating a record with auto-generated ID: Each execution creates a new record.
• Sending an email: Each execution sends another email.

Making non-idempotent operations safe with idempotency keys:

The caller generates a unique key for each logical operation and sends it with the request. The server checks if that key has been seen before. If yes, it returns the stored result without re-executing. This was covered briefly in Section 4 (Slide 56), but in the context of sagas it becomes critical.

Why idempotency is critical for sagas:

In a saga, the orchestrator sends a command to the Payment Service: “Authorize $59.98 for order ord_123.” If the Payment Service processes it but the response is lost, the orchestrator retries. Without idempotency, the customer is charged twice. With an idempotency key, the Payment Service recognizes the duplicate and returns the original result.

Interview Relevance: Idempotency is one of the most practical and frequently tested concepts in distributed systems interviews. If you propose retries anywhere in your design, the interviewer will ask how you handle duplicates.

SLIDE 98: At-Least-Once Delivery and the Exactly-Once Myth

Most message brokers guarantee at-least-once delivery. This means every message will be delivered, but some messages might be delivered more than once. Understanding why exactly-once delivery is effectively impossible helps you design systems correctly.

Why exactly-once is a myth in distributed systems:

Imagine the message broker delivers a message to the consumer. The consumer processes it and sends an acknowledgment. But the acknowledgment is lost due to a network failure. The broker thinks the message was not delivered and redelivers it. The consumer has already processed it. From the broker’s perspective, it delivered twice. From the consumer’s perspective, it received twice. Neither side did anything wrong.

The three delivery guarantees:

• At-most-once: The message is delivered zero or one times. If delivery fails, the message is lost. Fast but unreliable. Acceptable for metrics and analytics where losing a data point is not critical.
• At-least-once: The message is delivered one or more times. No messages are lost, but duplicates are possible. This is the default for most systems (Kafka, RabbitMQ, SQS).
• Effectively-once: At-least-once delivery combined with idempotent processing on the consumer side. Each message is delivered at least once, but the consumer’s idempotency logic ensures it is only processed once. This is what production systems actually achieve.

The practical approach: Design for at-least-once delivery (the default) and make every consumer idempotent (using inbox tables or idempotency keys). This gives you “effectively-once” processing, which is the strongest guarantee achievable in practice.

Interview Relevance: If an interviewer asks about “exactly-once delivery,” the correct answer is that it is not achievable in a distributed system, but you can achieve effectively-once semantics through idempotent consumers. This shows real understanding of distributed systems theory.

SLIDE 99: Dead-Letter Queues

A dead-letter queue (DLQ) is a holding area for messages that cannot be processed after multiple retry attempts. Instead of retrying forever or silently dropping the message, failed messages are moved to the DLQ for investigation.

When a message goes to the DLQ:

• The consumer has retried processing 3-5 times and keeps failing
• The message payload is malformed and cannot be deserialized
• A bug in the consumer causes a crash every time it encounters this specific message type
• A downstream dependency is permanently unavailable for this specific request

What happens to messages in the DLQ:

1. An alert fires notifying the on-call engineer that messages are accumulating in the DLQ
2. The engineer investigates: is this a bug, a bad message, or a temporary downstream outage?
3. If the issue is fixed (bug patched, downstream recovered), messages are replayed from the DLQ back to the main queue
4. If the message is genuinely unprocessable (corrupted data, invalid state), it is logged and archived

Why DLQs are essential for sagas:

In a saga, a compensating transaction that cannot be processed is a serious consistency problem. If “release inventory for order ord_123” fails and goes to the DLQ, inventory remains reserved for an order that has been cancelled. The DLQ ensures this failure is visible and eventually handled, rather than silently ignored.

Interview Relevance: Mentioning dead-letter queues when discussing message-driven architectures shows the interviewer you think about what happens when processing fails, not just when it succeeds.

SLIDE 100: Retry and Backoff

Retrying failed operations is essential in distributed systems, but retrying incorrectly can turn a minor outage into a catastrophic one.

The retry storm problem:

The Inventory Service goes down for 10 seconds. During those 10 seconds, 5,000 messages accumulate. All 5,000 messages retry simultaneously. The Inventory Service comes back up and immediately receives 5,000 requests. It cannot handle the burst and goes down again. This cycle repeats, and the service never recovers.

Exponential backoff:

Instead of retrying immediately, wait an increasing amount of time between retries: 1 second, 2 seconds, 4 seconds, 8 seconds. This spreads retries over time and gives the recovering service breathing room.

Jitter:

Even with exponential backoff, if 5,000 callers all start with a 1-second delay, they all retry at the same time. Adding random jitter (a random delay of 0 to backoff_interval) desynchronizes the retries so the recovering service receives a steady trickle instead of a burst.

Retry budget:

Set a maximum number of retries (typically 2-3 for synchronous calls, 5-10 for asynchronous messages). After exhausting the retry budget, fail the operation and move the message to a dead-letter queue.

The formula:

delay = min(base_delay * 2^attempt + random(0, base_delay), max_delay)

With base_delay = 1 second and max_delay = 30 seconds: attempt 1 waits 1-2s, attempt 2 waits 2-3s, attempt 3 waits 4-5s, attempt 4 waits 8-9s, capped at 30 seconds.

Interview Relevance: Mentioning “exponential backoff with jitter” is the expected answer whenever retries are discussed. If you just say “retry after 1 second,” the interviewer will ask about retry storms.

SLIDE 100B: Note It Down

Messaging reliability patterns to memorize:

1. Transactional outbox - Write events to the database, not the broker. Same transaction as the business data. Relay publishes later.
2. Inbox pattern - Consumer deduplicates by tracking processed event IDs in a local table.
3. Idempotency keys - Caller-generated unique key per operation. Server returns stored result on duplicate.
4. At-least-once delivery - Default for Kafka, RabbitMQ, SQS. Messages are never lost but may be duplicated.
5. Effectively-once - At-least-once delivery + idempotent consumers. The strongest practical guarantee.
6. Dead-letter queue - Failed messages go to a DLQ after retry budget is exhausted. Alert and investigate.
7. Exponential backoff with jitter - Prevent retry storms. Delay increases exponentially. Jitter desynchronizes retries.

The chain of reliability: Outbox ensures events are published. At-least-once delivery ensures events reach consumers. Inbox/idempotency ensures events are processed exactly once. DLQ ensures failures are visible. Backoff ensures retries do not cause cascading failure.

Interview Relevance: Being able to describe this full reliability chain in an interview demonstrates production-grade distributed systems knowledge. Most candidates know one or two of these patterns. Knowing all seven and how they compose is a differentiator.

SLIDE 101: Partial Failure Handling

Partial failure is the defining challenge of distributed systems. Unlike a monolith where an operation either fully succeeds or fully fails, microservices can leave you in a state where some steps succeeded and others did not.

The three response strategies for partial failure:

Strategy 1: Forward recovery (retry and complete)

If step 3 of 4 fails, retry step 3. If the failure is transient (network timeout, temporary unavailability), retrying may succeed and the saga completes normally. This is the first strategy to try and handles the majority of failures in practice.

Strategy 2: Backward recovery (compensate and cancel)

If step 3 fails permanently (insufficient funds, invalid data), run compensating transactions for steps 1 and 2. The saga moves backward, undoing completed steps and leaving the system in a clean cancelled state.

Strategy 3: Best-effort with human intervention

If compensation itself fails (the Inventory Service is down and cannot release reserved stock), the system logs the failure, alerts the operations team, and the saga enters a COMPENSATION_FAILED state. A human operator or automated recovery job resolves the inconsistency when the downstream service recovers.

Choosing between strategies:

• Transient error (timeout, 503)? → Retry with backoff (forward recovery)
• Business error (insufficient funds, out of stock)? → Compensate (backward recovery)
• Persistent infrastructure error? → DLQ and human intervention

Interview Tip: In an interview, explicitly mention all three strategies. “First, I retry because most failures are transient. If the failure is permanent, I run compensating transactions. If compensation fails, the saga enters a stuck state and the operations team is alerted.” This shows you think about the full failure spectrum.

Interview Relevance: The ability to describe what happens when things go wrong at multiple levels, not just the happy path, is the single biggest differentiator between senior and mid-level interview answers.

SLIDE 102: State Machines for Service Lifecycles

A state machine defines every possible state a business entity can be in and every valid transition between states. In microservices with sagas, state machines prevent invalid state transitions and make the system’s behavior predictable and debuggable.

Example: Order lifecycle state machine

Why state machines matter for sagas:

Without a state machine, an order can end up in an undefined state. What happens if the Payment Service sends a “PaymentAuthorized” event for an order that is already CANCELLED? Without a state machine, the Order Service might process it and move the order to PAYMENT_AUTHORIZED, overwriting the cancellation. With a state machine, the transition from CANCELLED to PAYMENT_AUTHORIZED is not defined, so it is rejected.

Implementation:

VALID_TRANSITIONS = {
    "CREATED": ["INVENTORY_RESERVED", "CANCELLED"],
    "INVENTORY_RESERVED": ["PAYMENT_AUTHORIZED", "CANCELLED"],
    "PAYMENT_AUTHORIZED": ["CONFIRMED", "CANCELLED"],
    "CONFIRMED": ["SHIPPED", "CANCELLED"],
    "SHIPPED": ["DELIVERED"],
    "CANCELLED": [],
    "DELIVERED": [],
}

def transition_order(order, new_status):
    if new_status not in VALID_TRANSITIONS.get(order.status, []):
        raise InvalidStateTransitionError(order.status, new_status)
    order.status = new_status
    db.save(order)

Every state change goes through this validation. Invalid transitions are rejected and logged, making bugs immediately visible instead of silently corrupting data.

Interview Relevance: Drawing a state machine diagram on the whiteboard when designing a saga shows the interviewer you think about every possible state, including error states. This is a hallmark of production-experienced engineers.

SLIDE 103: Worked Example, Checkout Saga (Happy Path)

Scenario: A customer places an order for 2 items totaling $59.98 on an e-commerce platform like Amazon. The checkout saga uses orchestration.

Saga steps (happy path):

Each step and its data:

Notice: Every mutating step has an idempotency key. The notification is the last step and is asynchronous, because you cannot unsend an email, and email failure should never block an order.

Interview Relevance: Being able to walk through a complete checkout saga with specific IDs, idempotency keys, and state transitions demonstrates production-level understanding of distributed workflows.

SLIDE 104: Worked Example, Checkout Saga (Failure Cases)

Using the same checkout saga from the previous slide, here is what happens when each step fails.

Failure Case 1: Inventory reservation fails (out of stock)

Failure Case 2: Payment authorization fails (insufficient funds)

Failure Case 3: Shipping creation fails after payment

Key insight: Not every failure triggers full compensation. Shipping failure after payment does not require refunding the customer. The correct response is to retry shipping, because the customer wants their order, not a refund. Full compensation (refund + cancel) is only appropriate when the core operation cannot proceed (no stock, no payment).

Interview Relevance: Walking through these specific failure cases and explaining why the response differs for each one is the highest-signal demonstration of saga understanding in an interview.

SLIDE 104B: Interview Practice

Question:

An interviewer asks: “In your checkout saga, the payment authorization succeeds, but then the Inventory Service goes down and you cannot release inventory during compensation. The customer has been charged but inventory is stuck in reserved state. What do you do?”

Take 60 seconds to think about this edge case before reading below.

Weak Answer:

“I would retry the inventory release until it succeeds.”

Why this is weak: The Inventory Service might be down for hours. The customer has been charged. Retrying indefinitely without any fallback leaves the customer in a bad state.

Strong Answer:

“This is a compensation failure, which is the hardest case in saga design. Here is how I would handle it.

First, the orchestrator retries the inventory release with exponential backoff and jitter, with a retry budget of 5 attempts over about 2 minutes. Most Inventory Service outages are transient and resolve within that window.

If retries are exhausted, the orchestrator marks the saga as COMPENSATION_FAILED and the failed compensation message goes to a dead-letter queue. An alert fires to the operations team.

For the customer, I would void the payment authorization immediately. Payment authorization is a hold on funds, not a charge. Voiding it costs nothing and releases the hold on the customer’s card. I do not need the Inventory Service to be available to void the payment.

For the stuck inventory reservation, when the Inventory Service recovers, a reconciliation job processes all DLQ messages and releases the orphaned reservations. This can also be done manually by the operations team if urgency requires it.

The saga’s final state is: order CANCELLED, payment voided, inventory released (eventually). The customer is never charged for a cancelled order.”

What makes this strong: The candidate addresses the immediate customer impact (void the payment hold), the eventual consistency resolution (DLQ and reconciliation), and the monitoring path (alerts to operations). They demonstrate that compensation failures are expected and planned for, not surprising edge cases.

SLIDE 105: Human Intervention and Manual Recovery

No matter how well you design your sagas, some failures cannot be resolved automatically. Acknowledging this and designing for human intervention is a sign of production maturity, not a sign of failure.

When human intervention is needed:

• A compensating transaction has failed after all retries and the DLQ contains messages that cannot be replayed because the downstream service has a data corruption issue
• A third-party payment provider charged the customer but returned an error response, so your system thinks the payment failed but the customer sees a charge on their card
• An inventory discrepancy exists where the physical warehouse count does not match the digital count, and automated reconciliation cannot resolve it
• A regulatory or compliance issue requires manual review before proceeding (fraud suspicion, sanctions screening)

Designing for human intervention:

• Saga status dashboard: A UI that shows all sagas, their current state, and any sagas in COMPENSATION_FAILED state. Operations teams can view the saga history, see which step failed, and trigger manual resolution.
• Manual compensation actions: Admin endpoints that allow operators to manually release inventory, void payments, or update order status. These are protected by authentication and audit logging.
• Runbooks: Step-by-step instructions for each type of compensation failure. “If the Payment Service charged the customer but the saga shows PAYMENT_FAILED, verify with the payment provider, then manually update the saga state.”

The honest reality:

In a system processing millions of orders per year, a small fraction (0.001-0.01%) will require human intervention. At 10 million orders per year, that is 100-1,000 cases annually. Building the tooling to handle these efficiently is part of production-grade microservices design.

Interview Relevance: Mentioning human intervention as a deliberate part of your saga design shows the interviewer you have production experience. Systems that claim to handle every failure automatically are either lying or have not encountered enough failure modes yet.

SLIDE 106: Section Summary and Interview Tips

Section 6 Summary:

Common Interview Mistakes:

Key Numbers to Remember:

How This Connects to Other Sections:

Section 3 (Service Decomposition) defined aggregates as the consistency boundary within a service. Everything inside an aggregate is a local ACID transaction. Everything that crosses an aggregate boundary is a saga, which this section taught you how to design. Section 4 (Communication Patterns) provided the messaging infrastructure (events, queues, streams) that sagas rely on. The outbox pattern from this section ensures reliable event publishing over the messaging infrastructure from Section 4. Section 5 (Data Ownership) identified cross-service writes as the hardest data challenge. This section provided the solution: sagas with compensating transactions, orchestrated or choreographed, with the full reliability chain of outbox, inbox, idempotency, and DLQ. Section 7 (API Gateway) handles the edge where checkout requests arrive, and Section 8 (Reliability) covers the observability needed to monitor saga health in production.