INTER-SERVICE COMMUNICATION

SLIDE 44: Section Intro

Section 4: Inter-Service Communication

Once you have drawn service boundaries (Section 3), the next question is immediate: how do these services talk to each other? The answer to this question is not a technology choice. It is an architecture decision that determines coupling, latency, failure behavior, and consistency across your entire system.

Choosing REST vs gRPC vs message queues is not about which technology is “better.” It is about which communication pattern matches the coupling and consistency requirements of each specific interaction. A system typically uses multiple patterns for different interactions.

What We Will Cover:

1. Synchronous vs asynchronous communication and the trade-offs of each
2. REST, gRPC, and GraphQL for synchronous communication
3. Message queues and event streams for asynchronous communication
4. Commands vs events and why the distinction matters
5. Resilience patterns: timeouts, retries, circuit breakers, bulkheads, and backpressure
6. Contract versioning and schema evolution for independent deployment
7. Correlation IDs for tracing requests across services
8. A decision framework for choosing the right pattern

Where This Appears in Real Systems:

In a platform like Uber, the Rider App uses REST to request a ride. The Dispatch Service uses gRPC internally for low-latency driver matching. The Trip Service publishes events to a stream (like Kafka) that the Payment Service, Analytics Service, and Notification Service consume independently. A single ride request uses at least three different communication patterns. This section teaches you when and why to use each.

SLIDE 45: Synchronous vs Asynchronous Communication

Every inter-service communication falls into one of two categories, and the choice between them is the most important communication decision you will make.

Synchronous communication:

Service A sends a request to Service B and waits for a response before continuing. The caller is blocked until the response arrives or a timeout occurs.

Asynchronous communication:

Service A sends a message and continues immediately without waiting for a response. Service B processes the message whenever it is ready.

The fundamental trade-off:

• Synchronous gives you immediate feedback (success/failure) but creates temporal coupling. If Service B is down, Service A fails too.
• Asynchronous removes temporal coupling (Service A does not depend on Service B being available right now) but you lose immediate feedback. You do not know if Service B processed the message successfully until later.

A practical rule: Use synchronous communication when the caller needs the response to continue its work. Use asynchronous communication when the caller does not need to wait for the result.

Interview Relevance: The first communication decision in any interview should be sync vs async for each interaction. Starting here shows you are thinking about coupling and failure modes, not just picking a technology.

SLIDE 46: REST

REST (Representational State Transfer) is the most common communication protocol for microservices. It uses HTTP and is the default choice for public APIs and straightforward request-response interactions.

When REST fits well:

• Public-facing APIs where broad client compatibility matters (browsers, mobile apps, third-party integrations)
• CRUD-style operations (create, read, update, delete) on resources
• Interactions where simplicity and widespread tooling support matter more than raw performance

A typical REST interaction:

POST /orders HTTP/1.1
Host: order-service.internal
Content-Type: application/json

{
  "customer_id": "cust_12345",
  "items": [
    {"product_id": "prod_789", "quantity": 2}
  ]
}

HTTP/1.1 201 Created
Content-Type: application/json

{
  "order_id": "ord_98765",
  "status": "CREATED",
  "total": 59.98
}

REST limitations in microservices:

• Over-fetching: The client receives all fields even if it only needs two. A mobile app displaying an order summary gets the full order object with 30 fields.
• Chattiness: Displaying a product page might require calling the Catalog Service, Pricing Service, Inventory Service, and Reviews Service separately. Four HTTP round trips.
• No streaming: Standard REST is request-response. It does not support server-to-client streaming or bidirectional communication natively.
• Text-based overhead: JSON payloads are human-readable but larger than binary formats. For internal service-to-service calls at high throughput, this overhead adds up.

Interview Relevance: REST is the safe default for any external-facing API. For internal service-to-service calls, explain when you would use it and when you would choose gRPC instead.

SLIDE 47: gRPC

gRPC is a high-performance RPC framework that uses Protocol Buffers (protobuf) for serialization and HTTP/2 for transport. It is designed for internal service-to-service communication where performance and type safety matter.

When gRPC fits well:

• Internal service-to-service calls where both ends are under your control
• High-throughput, low-latency communication (10x-100x more calls per second than REST)
• Strongly typed contracts where you want compile-time guarantees that the request and response structures are correct
• Streaming use cases (server streaming, client streaming, bidirectional streaming)

How a gRPC contract is defined:

service OrderService {
  rpc CreateOrder(CreateOrderRequest) returns (CreateOrderResponse);
  rpc GetOrder(GetOrderRequest) returns (Order);
  rpc StreamOrderUpdates(GetOrderRequest) returns (stream OrderUpdate);
}

message CreateOrderRequest {
  string customer_id = 1;
  repeated OrderItem items = 2;
}

message OrderItem {
  string product_id = 1;
  int32 quantity = 2;
}

gRPC vs REST:

• Protobuf messages are binary, roughly 5-10x smaller than equivalent JSON
• HTTP/2 supports multiplexing (multiple requests over one connection) and header compression
• Code generation produces typed client and server stubs in any language. No hand-written HTTP parsing.
• The trade-off: gRPC is harder to debug (binary payloads are not human-readable), harder to call from browsers without a proxy, and requires both sides to share the proto definition file

Interview Tip: A strong pattern in interviews is “REST for external APIs, gRPC for internal service-to-service calls.” This shows you understand that the right protocol depends on who the consumer is, not which technology is newer.

Interview Relevance: Mentioning gRPC for internal communication shows the interviewer you think about performance and type safety at scale, not just developer convenience.

SLIDE 48: GraphQL

GraphQL is a query language for APIs that lets the client specify exactly which fields it needs. It is not a replacement for REST or gRPC. It serves a different purpose.

When GraphQL fits well:

• Client-facing APIs where different clients (web, iOS, Android) need different subsets of the same data
• Aggregation layers that combine data from multiple backend services into one client response
• Situations where over-fetching and under-fetching from REST APIs are causing performance problems on the client side

How GraphQL solves the over-fetching problem:

A mobile app displaying an order summary only needs order ID, status, and total. With REST, it receives the full order object with 30 fields. With GraphQL, the client requests exactly what it needs:

query {
  order(id: "ord_98765") {
    orderId
    status
    total
  }
}

The response contains only those three fields. No wasted bandwidth, no unnecessary data processing on the client.

Where GraphQL fits in a microservices architecture:

GraphQL works best as an edge layer, not as a service-to-service protocol. A GraphQL gateway sits between clients and backend microservices. It translates client queries into calls to the appropriate backend services (which use REST or gRPC internally) and assembles the response.

GraphQL risks:

• A single complex query can trigger expensive joins across multiple backend services. Without query complexity limits, clients can accidentally (or maliciously) overload the system.
• The GraphQL gateway can become a “god gateway” that accumulates business logic. It should only translate and aggregate, never contain business rules.
• Caching is harder than REST because every query can be unique.

Interview Relevance: Mentioning GraphQL as an edge aggregation layer (not as a service-to-service protocol) shows you understand where it adds value and where it creates risk.

SLIDE 49: REST vs gRPC vs GraphQL

The typical microservices pattern:

• REST for public-facing APIs and third-party integrations
• gRPC for internal service-to-service calls where performance matters
• GraphQL as an optional edge layer for mobile/web clients that need flexible queries
• Many systems use REST and gRPC together, with no GraphQL layer at all

Interview Relevance: An interviewer does not want to hear “I would use REST.” They want to hear “I would use REST for the public API because it has broad client compatibility, and gRPC for internal calls between the Order Service and Payment Service because the latency budget is tight and we want typed contracts.”

SLIDE 49B: Interview Practice

Question:

You are designing the communication between services in a food delivery platform. The interviewer asks: “How would the Rider App communicate with your backend, and how would your backend services communicate with each other?”

Take 60 seconds to think about your answer before reading below.

Weak Answer:

“Everything uses REST. It is simple, well-understood, and easy to implement.”

Why this is weak: Using REST for everything ignores the different requirements of external vs internal communication. Internal calls between the Dispatch Service and Driver Location Service happen thousands of times per second with a sub-10ms latency budget. REST with JSON serialization adds unnecessary overhead for these calls.

Strong Answer:

“I would use different protocols for different interactions based on their requirements.

For the Rider App to backend communication, I would use REST through an API Gateway. REST gives us broad compatibility across iOS, Android, and web clients, and the API Gateway handles authentication, rate limiting, and request routing.

For internal service-to-service calls, I would use gRPC. The Dispatch Service needs sub-10ms responses from the Driver Location Service when matching a rider with nearby drivers. gRPC’s binary serialization and HTTP/2 transport give us the performance we need, and the proto definitions give us compile-time type safety between teams.

For communication that does not need immediate responses, I would use asynchronous events. When a trip is completed, the Trip Service publishes a trip.completed event. The Payment Service consumes it to charge the rider. The Analytics Service consumes it to update dashboards. The Notification Service consumes it to send the receipt. None of these need to block the trip completion flow.”

What makes this strong: The candidate matches each communication pattern to a specific requirement (client compatibility, low latency, decoupling) and explains why the choice fits.

SLIDE 50: Message Queues

A message queue is a buffer between a producer (the service sending a message) and a consumer (the service processing it). The producer puts a message on the queue and moves on. The consumer picks it up when ready.

Key characteristic: Each message is delivered to exactly one consumer. If three workers are consuming from the same queue, each message goes to only one of them. This makes queues ideal for task distribution.

When to use message queues:

• Background processing - User uploads a video. The API returns immediately. A worker picks up the encoding task from the queue and processes it asynchronously.
• Load leveling - The Order Service receives 10,000 orders per minute during a flash sale. The Payment Service can only handle 2,000 per minute. The queue absorbs the burst, and the Payment Service processes at its own pace.
• Decoupling - The Order Service does not need to know which service processes payments. It puts a message on the queue. The Payment Service (or any replacement) consumes it.

Real-world example: In a platform like Amazon, when you click “Place Order,” the API returns immediately with your order confirmation. The actual payment processing, inventory reservation, and shipping creation happen asynchronously via messages. This is why your order status says “Processing” for a few seconds before updating.

Interview Relevance: Message queues are the right answer whenever the caller does not need an immediate response and the work can be processed in the background. Using synchronous calls for background work is a common interview mistake.

SLIDE 51: Event Streams

An event stream is a durable, ordered log of events that multiple consumers can read independently. Unlike a message queue where each message goes to one consumer, an event stream allows every consumer to read every event.

Key characteristic: Events are persisted in the stream. Consumers maintain their own read position (offset). Each consumer reads at its own pace. Adding a new consumer does not affect existing ones. You can even replay events from the beginning of the log.

When to use event streams:

• Multiple consumers need the same event - An “order placed” event is relevant to payments, inventory, shipping, analytics, and notifications. A queue would deliver it to only one of them.
• Event history matters - A new Analytics Service is deployed and needs to process all historical order events to build its initial dashboard. It reads the stream from the beginning.
• Event-driven architecture - Services react to events rather than being told what to do. The Payment Service does not receive a command “process this payment.” It observes an event “an order was placed” and decides to process the payment.

Real-world example: A platform like LinkedIn uses event streams (Kafka) to propagate profile updates. When you update your job title, that event is consumed by the search index (to update search results), the feed service (to notify your connections), the recommendation engine (to update job suggestions), and the analytics service (to track engagement). Each consumer processes the event independently.

Interview Relevance: Event streams are the right answer when multiple services need to react to the same event independently. This is one of the most common patterns in large-scale microservices.

SLIDE 52: Commands vs Events

This distinction is subtle but critical. Getting it wrong creates tight coupling between services even when using asynchronous messaging.

Commands: “Do this”

A command is an instruction directed at a specific service. The sender knows who will process it and expects a specific outcome.

Example: “ProcessPayment(order_id=123, amount=59.98)” is sent to the Payment Service. The sender expects the payment to be processed.

Events: “This happened”

An event is a notification that something occurred. The publisher does not know or care who consumes it. It does not expect any specific outcome.

Example: “OrderPlaced(order_id=123, customer_id=456, items=[…])” is published to a stream. The publisher has no idea which services will react to it.

Why this matters for coupling:

• Commands create coupling. The Order Service must know that the Payment Service exists and what API it exposes. If the Payment Service is renamed, relocated, or replaced, the Order Service must be updated.
• Events reduce coupling. The Order Service publishes “OrderPlaced” without knowing that the Payment Service, Notification Service, and Analytics Service all consume it. New consumers can be added without changing the Order Service.

When to use each:

• Use commands when you need a guaranteed outcome from a specific service (e.g., “authorize this payment before confirming the order”)
• Use events when you want to notify the system that something happened and let interested services react independently (e.g., “an order was placed, whoever cares can act on it”)

Interview Tip: When describing your communication patterns in an interview, use the correct terminology. “The Order Service publishes an OrderPlaced event” sounds fundamentally different from “the Order Service sends a ProcessPayment command to the Payment Service.” The first is loosely coupled. The second is tightly coupled. Interviewers notice this distinction.

Interview Relevance: Using “commands” and “events” correctly in an interview signals that you understand coupling at the message level, not just at the service level.

SLIDE 53: Message Queues vs Event Streams

Choosing between them:

• If only one consumer should process each message (background jobs, work distribution), use a message queue.
• If multiple consumers need the same event (broadcasting to payment, analytics, notifications), use an event stream.
• If you need to replay historical events (rebuilding read models, onboarding new services), use an event stream.
• Many systems use both. Message queues for specific task routing. Event streams for system-wide event broadcasting.

Interview Relevance: Knowing the difference between a message queue and an event stream, and choosing the right one for each interaction, is a specific skill interviewers test for.

SLIDE 54: Request Choreography and the Fan-Out Problem

When a single user action triggers calls to multiple downstream services, you face the fan-out problem. How you handle it determines your system’s latency and resilience.

The fan-out problem:

A single “place order” request triggers 5 downstream calls. If each call takes 50ms, and they happen sequentially, total latency is 250ms. If the Fraud Service is slow (200ms), the entire order takes 400ms.

Strategies to manage fan-out:

Parallelize independent calls. Inventory check, pricing calculation, and fraud check can happen simultaneously. Only payment authorization must wait for the fraud check result. Parallel execution reduces latency from the sum of all calls to the duration of the longest parallel branch.

Make non-critical calls asynchronous. Notification does not need to complete before the order is confirmed. Publish an event and let the Notification Service process it later. This removes it from the critical path entirely.

Cache frequently needed data. If the Order Service calls the Pricing Service for every order and pricing data changes infrequently, cache pricing data locally. This eliminates a network call for most requests.

Set per-hop timeout budgets. If the total request budget is 500ms, allocate 100ms to inventory, 100ms to pricing, 200ms to payment (including fraud), and 100ms for overhead. If any hop exceeds its budget, fail fast rather than waiting.

Interview Relevance: When an interviewer sees a fan-out in your design, they will ask how you handle it. Having a strategy (parallelize, async non-critical, cache, timeout budgets) ready is essential.

SLIDE 54B: Interview Practice

Question:

You are designing the checkout flow for an e-commerce platform. When a user clicks “Place Order,” your Order Service needs to: verify inventory, calculate the final price (with discounts and tax), authorize payment, create a shipment record, and send a confirmation email. The interviewer asks: “Walk me through the communication pattern for this flow.”

Take 60 seconds to design the flow before reading below.

Weak Answer:

“The Order Service calls each downstream service one at a time: first Inventory, then Pricing, then Payment, then Shipping, then Notification. Each call is REST.”

Why this is weak: Sequential synchronous calls create maximum latency (sum of all calls) and maximum fragility (any one failure blocks everything). Notification has no business being on the critical path, it does not affect whether the order succeeds.

Strong Answer:

“I would split these into critical-path synchronous calls and non-critical asynchronous events.

On the critical path, I would parallelize where possible. Inventory check and pricing calculation are independent, so they run in parallel. Both must succeed before we proceed. Then payment authorization happens synchronously because we need to know if payment succeeded before confirming the order.

Shipment creation is important but does not need to block the order confirmation. The Order Service publishes an OrderConfirmed event, and the Shipping Service consumes it to create the shipment asynchronously. If shipment creation fails, it retries from the event.

Notification is fully asynchronous. The Order Service publishes the same OrderConfirmed event, and the Notification Service consumes it to send the confirmation email. If the email fails, the order is still valid.

This gives us a critical path of: max(inventory_check, pricing_calculation) + payment_authorization. If inventory and pricing each take 50ms and payment takes 100ms, total checkout latency is about 150ms instead of 300ms for sequential calls.”

What makes this strong: The candidate separates critical from non-critical, parallelizes independent calls, uses async for fire-and-forget steps, and provides specific latency math.

SLIDE 55: Timeout Strategy

Every network call in a microservices system must have a timeout. A call without a timeout can hang indefinitely, holding resources (threads, connections, memory) that other requests need.

The problem without timeouts:

The Payment Service becomes slow (responding in 30 seconds instead of 100ms). The Order Service waits 30 seconds per request. Order Service threads pile up waiting. New orders cannot be processed because all threads are blocked. The Order Service effectively goes down because the Payment Service is slow. This is a cascading failure.

Per-hop timeout budgets:

If the total request budget from client to response is 2 seconds, each downstream call gets a fraction:

• Inventory check: 200ms timeout
• Pricing calculation: 200ms timeout
• Payment authorization: 500ms timeout (external provider, slower)
• Total overhead (serialization, network, processing): 300ms
• Remaining buffer: 800ms

If the Inventory Service does not respond within 200ms, fail immediately and return an error rather than consuming 1.8 seconds of remaining budget waiting.

Timeout guidelines:

• P99 latency of the downstream service is a good starting point for the timeout value
• Set timeouts on every outgoing call, including database calls and cache calls
• Log when timeouts are hit so you can detect degraded services
• Never use “no timeout” or “infinite timeout” as a default

Interview Relevance: Mentioning timeouts proactively when describing service communication shows the interviewer you understand production realities, not just happy-path design.

SLIDE 56: Retry Strategy and Idempotency

When a call fails or times out, the natural instinct is to retry. But retrying incorrectly can make a bad situation catastrophically worse.

When retrying is safe:

A retry is safe only when the operation is idempotent, meaning executing it twice produces the same result as executing it once. Reading data is always idempotent. Creating a resource with a client-generated ID is idempotent (creating order “ord_123” twice results in one order). Incrementing a counter is not idempotent (incrementing twice gives a different result than incrementing once).

Idempotency keys:

For operations that are not naturally idempotent, attach a unique idempotency key to each request. The server stores the key and the result. If the same key arrives again, the server returns the stored result without re-executing.

def authorize_payment(idempotency_key, order_id, amount):
    existing_result = db.get_by_idempotency_key(idempotency_key)
    if existing_result:
        return existing_result

    result = payment_gateway.charge(order_id, amount)
    db.save_with_idempotency_key(idempotency_key, result)
    return result

Retry with exponential backoff and jitter:

If all callers retry after exactly 1 second, the failing service gets hit with a synchronized wave of retries. Add exponential backoff (wait 1s, 2s, 4s, 8s) and random jitter (add a random 0-500ms) so retries are spread out.

Retry limits: Set a maximum number of retries (typically 2-3). After that, fail and let the caller handle the error. Unlimited retries can overload a recovering service.

Interview Relevance: Explaining idempotency keys and exponential backoff with jitter demonstrates practical distributed systems knowledge that goes beyond textbook concepts.

SLIDE 57: Circuit Breakers

A circuit breaker prevents a service from repeatedly calling a downstream dependency that is failing. It is named after electrical circuit breakers that cut power to prevent damage.

The three states:

How it works:

• Closed (normal): Requests pass through to the downstream service. The circuit breaker counts failures. When failures exceed a threshold (e.g., 5 failures in 10 seconds), the breaker trips to Open.
• Open (protecting): All requests fail immediately without calling the downstream service. This gives the failing service time to recover and prevents the caller from wasting resources on calls that will fail. After a timeout (e.g., 30 seconds), the breaker moves to Half-Open.
• Half-Open (testing): One test request is sent to the downstream service. If it succeeds, the breaker returns to Closed. If it fails, the breaker returns to Open for another timeout period.

Why this matters:

Without a circuit breaker, the Order Service keeps sending requests to a failing Payment Service, waiting for timeouts on each one. With a circuit breaker, after 5 failures the Order Service stops calling the Payment Service immediately (no timeout wait), and returns a graceful error to the user: “Payment is temporarily unavailable, please try again shortly.”

Interview Relevance: Circuit breakers are one of the most frequently asked resilience patterns in system design interviews. Know the three states, the failure threshold, and the recovery mechanism.

SLIDE 58: Bulkheads and Backpressure

These two patterns prevent failures from spreading across the system.

Bulkheads:

Named after the watertight compartments in a ship’s hull. If one compartment floods, the others stay dry. In microservices, bulkheads isolate failure domains so that a problem with one dependency does not exhaust resources needed for other operations.

Example: The Order Service calls both the Payment Service and the Recommendation Service. Without bulkheads, both calls share the same thread pool (100 threads). If the Recommendation Service becomes slow and 90 threads are stuck waiting on it, only 10 threads remain for payment processing. Payments slow down because of recommendations.

With bulkheads, you assign separate thread pools: 70 threads for payments (critical) and 30 for recommendations (non-critical). If recommendations become slow, only the recommendation thread pool is exhausted. Payments continue unaffected.

Backpressure:

When a service receives more requests than it can handle, it must signal the caller to slow down rather than accepting everything and degrading for all users.

Backpressure mechanisms:

• Return HTTP 429 (Too Many Requests) when the service is at capacity
• Use bounded queues that reject messages when full instead of growing unbounded until memory runs out
• Implement rate limiting per caller so one noisy client cannot starve others

Without backpressure: A service accepts 50,000 requests per second when it can only handle 10,000. Response times go from 50ms to 5 seconds. All 50,000 callers get a degraded experience. With backpressure: The service accepts 10,000 and rejects 40,000 with a clear signal to retry later. The 10,000 accepted requests get normal performance.

Interview Relevance: Mentioning bulkheads and backpressure in an interview shows you think about failure isolation at a granular level, beyond just circuit breakers.

SLIDE 59: Resilience Patterns Summary

How they work together:

A well-designed service uses all of these patterns simultaneously. A request goes out with a timeout. If it fails, a retry is attempted with backoff (only if idempotent). If multiple retries fail, the circuit breaker trips. The dependency is isolated behind a bulkhead so other operations continue. The service applies backpressure if its own capacity is threatened.

Interview Relevance: Being able to explain how these patterns compose together, not just define each one in isolation, is what distinguishes a senior-level answer from a mid-level answer.

SLIDE 59B: Note It Down

Six resilience patterns to memorize:

1. Timeouts - Every call, no exceptions. Use P99 latency as starting point.
2. Retries - Only for idempotent operations. Exponential backoff + random jitter. Max 2-3 retries.
3. Circuit breakers - Three states: Closed, Open, Half-Open. Trips after failure threshold.
4. Bulkheads - Separate resource pools per dependency. Critical paths get more resources.
5. Backpressure - Reject excess load with HTTP 429. Bounded queues. Rate limiting.
6. Idempotency keys - Client-generated unique key. Server deduplicates. Required for safe retries on writes.

The sentence to remember:

“Every outgoing call has a timeout, retries only if idempotent with backoff and jitter, is protected by a circuit breaker, isolated by a bulkhead, and the service applies backpressure when overloaded.”

Interview Relevance: If you can state this sentence naturally during an interview and then explain any pattern the interviewer asks about, you have demonstrated production-grade distributed systems knowledge.

SLIDE 60: Contract Versioning and Backward Compatibility

In microservices, services deploy independently. This means version 2 of the Payment Service must work with version 1 of the Order Service, and vice versa. If a new deployment breaks existing consumers, you do not have independent deployment. You have a release train.

The backward compatibility rule:

You can add new fields. You can add new endpoints. You cannot remove fields that consumers depend on. You cannot change the type of existing fields. You cannot rename existing endpoints.

REST versioning strategies:

• URI versioning: /api/v1/orders and /api/v2/orders. Simple but requires routing logic and maintaining multiple versions.
• Header versioning: Accept: application/vnd.company.v2+json. Cleaner URLs but less visible.
• Backward-compatible evolution (preferred): Add new fields without removing old ones. Consumers that do not know about the new fields simply ignore them. No version number needed.

Example of backward-compatible change:

Original response: {"order_id": "123", "status": "CONFIRMED", "total": 59.98}

Updated response: {"order_id": "123", "status": "CONFIRMED", "total": 59.98, "currency": "USD", "estimated_delivery": "2024-03-15"}

Old consumers continue working because they ignore currency and estimated_delivery. New consumers can use the new fields. No version bump required.

Interview Relevance: When you say “services deploy independently” in an interview, the interviewer may ask “how do you ensure one service’s deployment does not break another?” Backward-compatible API evolution is the answer.

SLIDE 61: Schema Evolution

For services communicating via gRPC (protobuf) or event streams (Avro, JSON Schema), the schema defines the contract. Schema evolution rules determine which changes are safe to make without breaking consumers.

Protobuf evolution rules:

• Safe: Add a new optional field (assign a new field number). Existing consumers ignore the new field.
• Safe: Deprecate a field (stop populating it, but do not reuse the field number). Existing consumers that still read it get the default value.
• Unsafe: Remove a field and reuse its number. A consumer compiled against the old schema interprets the new field’s data as the old field’s type.
• Unsafe: Change a field’s type (e.g., int32 to string). Binary decoding breaks.

Avro evolution (commonly used with Kafka):

Avro uses a schema registry. Producers register their schema. Consumers fetch the schema and can handle both old and new formats. Avro supports three compatibility modes:

• Backward compatible: New schema can read data written with the old schema. You can add fields with defaults.
• Forward compatible: Old schema can read data written with the new schema. You can remove fields with defaults.
• Full compatible: Both backward and forward. The safest but most restrictive.

The practical rule: Always add fields with default values. Never remove fields in the same release that consumers depend on. Deprecate first, then remove after all consumers have migrated.

Interview Tip: If you propose gRPC or Kafka in an interview, the interviewer may ask “how do you handle schema changes?” Knowing the evolution rules for protobuf or Avro shows you have thought about the long-term maintainability of the system, not just the initial design.

Interview Relevance: Schema evolution is the mechanism that makes independent deployment possible when services share contracts. Understanding it separates theory from practice.

SLIDE 62: Consumer-Driven Contract Testing

Contract testing verifies that a service’s API continues to work for its consumers after a change. It catches breaking changes before deployment, not after.

The problem it solves:

The Payment Service team adds a new field and accidentally renames transaction_id to txn_id. Unit tests pass because they test the Payment Service in isolation. Integration tests in staging catch the break, but only after deployment. The Order Service, which depends on transaction_id, is now broken in staging.

How consumer-driven contract testing works:

1. Each consumer (Order Service, Refund Service) writes a contract describing what it expects from the Payment Service API: “I send a POST to /payments with these fields, and I expect a response containing transaction_id and status.”
2. These contracts are shared with the Payment Service team.
3. The Payment Service runs all consumer contracts as part of its CI pipeline.
4. If any contract fails, the build fails, and the team knows which consumers would break before deploying.

The key insight: The consumers define the contract, not the provider. This ensures that the provider never accidentally breaks a field or endpoint that a consumer depends on. Changes that no consumer cares about (adding a new field, adding a new endpoint) pass all contracts automatically.

Tools: Pact is the most widely used consumer-driven contract testing framework. It supports REST, gRPC, and messaging contracts.

Interview Relevance: Mentioning contract testing in an interview shows you think about the lifecycle of service communication, not just the initial implementation.

SLIDE 63: Correlation IDs and Request Tracing

In a monolith, debugging is straightforward. One request, one process, one log file, one stack trace. In microservices, a single user request can touch 5-10 services. Without a way to connect the logs from all those services, debugging is nearly impossible.

How correlation IDs work:

The API Gateway generates a unique correlation ID (e.g., a UUID) for every incoming request. This ID is passed to every downstream service in the request headers. Every service includes the correlation ID in all of its log entries for that request.

What this enables:

When a user reports “my order failed,” you search your log aggregation system for that correlation ID. You instantly see every log entry from every service that was involved in that request, in chronological order. You can trace exactly where the failure occurred.

Implementation requirements:

• The API Gateway generates the correlation ID (or accepts one from the client for mobile debugging)
• Every service reads the correlation ID from incoming headers and includes it in all outgoing calls
• Every log entry includes the correlation ID as a structured field
• A log aggregation system (like the ELK stack or Datadog) lets you search by correlation ID

Interview Relevance: When an interviewer asks “how would you debug a problem in this microservices system?”, correlation IDs and distributed tracing are the expected answer. Without them, you are guessing.

SLIDE 63B: Interview Practice

Question:

An interviewer presents this scenario: “We have 8 microservices. A user reports that their order was placed but they never received a confirmation email. The order exists in the database. How would you debug this?”

Take 60 seconds to think about your debugging approach before reading below.

Weak Answer:

“I would check the logs of the Notification Service to see if it tried to send the email.”

Why this is weak: It jumps to one service without understanding the full request path. Maybe the Order Service never published the order confirmation event. Maybe the event was published but the message queue lost it. Maybe the Notification Service consumed the event but the email provider was down. Checking one service’s logs in isolation does not tell you where in the chain the failure occurred.

Strong Answer:

“First, I would get the order ID from the user’s report and look up the correlation ID for that request in our logs. Using the correlation ID, I would trace the request through every service it touched.

I would check: did the Order Service successfully create the order? The user says yes and the order exists, so this step succeeded. Did the Order Service publish an OrderConfirmed event? I would check the Order Service logs for the correlation ID and look for an event publication log entry. If the event was published, I would check the event stream or message queue to verify it was persisted. Then I would check the Notification Service consumer logs to see if it received the event. If it did, I would check whether it called the email provider and what response it got.

This trace tells me exactly where the chain broke: was it the event not being published, the event not being delivered, the Notification Service failing to process it, or the email provider rejecting the send. Each scenario has a different fix.”

What makes this strong: The candidate uses correlation IDs and traces the full request path systematically, checking each link in the chain rather than guessing which service failed.

SLIDE 64: Communication Pattern Decision Framework

Use this framework to choose the right communication pattern for each interaction in your system.

Step 1: Does the caller need the response to continue?

• Yes, the response is required for the next step (e.g., “I need to know if payment was authorized before confirming the order”) → Synchronous
• No, the work can happen independently (e.g., “send a confirmation email whenever you get to it”) → Asynchronous

Step 2: If synchronous, which protocol?

• External clients (browsers, mobile, third parties) → REST
• Internal service-to-service with tight latency budget → gRPC
• Client needs flexible field selection across multiple backend services → GraphQL at the edge

Step 3: If asynchronous, queue or stream?

• Only one consumer should process each message (task distribution, background jobs) → Message queue
• Multiple consumers need the same event (broadcasting, event-driven architecture) → Event stream
• You need event replay capability (rebuilding state, onboarding new consumers) → Event stream

Step 4: What resilience patterns does this call need?

• Every synchronous call: timeout + circuit breaker + bulkhead
• Every retryable call: exponential backoff + jitter + idempotency key
• Every service: backpressure when overloaded

Interview Tip: Walking through this decision framework on the whiteboard, choosing different patterns for different interactions and explaining why, is one of the most effective ways to demonstrate communication design skill in an interview.

Interview Relevance: The framework itself is less important than the reasoning behind each choice. Interviewers want to see that you evaluate coupling, latency, failure modes, and consistency for each interaction, not apply one pattern everywhere.

SLIDE 65: Section Summary and Interview Tips

Section 4 Summary:

Common Interview Mistakes:

Key Numbers to Remember:

How This Connects to Other Sections:

Section 5 (Data Ownership) builds on the event-driven patterns from this section. When each service owns its data, cross-service data access often happens through event consumption and materialized views, not synchronous API calls. Section 6 (Sagas and Consistency) uses both commands and events to coordinate multi-step workflows across services. The orchestration pattern uses commands. The choreography pattern uses events. The choice between them maps directly to the coupling trade-offs covered in this section. Section 8 (Reliability and Evolution) expands the resilience patterns from this section into system-level failure isolation and observability.