Chapter 3: Stream Processing Patterns - Data Engineering Course: Complete Interview Preparation | PlayBook Library

SLIDE 37: Section Intro, Stream Processing Patterns

Section 4: Stream Processing Patterns

Stream processing is what you reach for when batch latency is not acceptable. When a credit card transaction needs to be checked for fraud within 500 milliseconds, when a ride-sharing app needs to update the driver’s ETA every 5 seconds, when an IoT sensor anomaly needs to trigger an alert before the equipment fails, batch processing cannot help. These problems demand continuous, event-at-a-time processing.

This section teaches the core concepts of stream processing: how Kafka works as the backbone of streaming architectures, how consumer groups enable parallel processing, how windowing lets you aggregate over time, and how to handle the messy reality of late-arriving and out-of-order data. By the end of this section, you will be able to design a streaming pipeline and defend every architectural decision in an interview.

Recognition Signal:

When the interviewer mentions real-time, low-latency, continuous processing, or event-driven data, you are in stream processing territory. Prompts like “Detect fraud in real-time,” “Build a live dashboard,” “Process IoT sensor data continuously,” or “Trigger alerts when anomalies occur” all require streaming.

What We’ll Cover:

When streaming is the right choice (and when it is overkill)
Apache Kafka: architecture, partitions, offsets, consumer groups, and delivery guarantees
Kafka vs traditional message queues and when each applies
Windowing patterns: tumbling, sliding, and session windows
Late-arriving data, watermarks, and the completeness-vs-latency trade-off
Stream processing frameworks: Kafka Streams, Flink, Spark Structured Streaming, Beam
Stateful stream processing: state stores, checkpointing, and state management

Connection to the Pipeline Mental Model:

Stream processing spans the Ingest and Transform stages of the five-stage pipeline. Kafka serves as both the ingestion mechanism (receiving events from producers) and the storage layer for in-flight data (the commit log retains events for configurable periods). Stream processors (Flink, Kafka Streams) read from Kafka, transform events, and write results to the Serve layer (a database, cache, or another Kafka topic). In Section 3, we covered batch transformation. This section covers its streaming counterpart.

SLIDE 38: When Streaming is the Right Choice

Streaming adds complexity and cost compared to batch. It is the right choice only when the business genuinely requires low-latency data processing. The discipline of asking “does this actually need to be real-time?” before proposing streaming is a senior-level skill.

Streaming is the right choice when:

Sub-second to sub-minute action is required. Fraud detection must block a transaction before it completes. A ride-sharing app must update pricing every few seconds. A live dashboard must reflect events within seconds of occurrence.
Events must trigger immediate downstream actions. An order placed must immediately decrement inventory, send a confirmation email, and notify the warehouse. Waiting for a batch job is not acceptable.
The data is naturally continuous and unbounded. IoT sensor readings, user clickstreams, and application logs are continuous streams with no natural “batch boundary.” Processing them as streams aligns with how the data arrives.
Time-sensitive pattern detection. Detecting a DDOS attack, a sudden spike in error rates, or unusual login patterns requires analyzing events as they arrive, not hours later.

Streaming is NOT the right choice when:

The consumer tolerates minutes or hours of latency. Daily reports, weekly aggregations, and ML model training do not need streaming.
The workload requires complex joins across large historical datasets. Joining a 500 GB fact table with a 200 MB dimension table is a batch problem. Stream processors are designed for small state and simple joins, not full table scans.
Cost efficiency is the priority and latency is not a constraint. Streaming infrastructure runs 24/7 and cannot use spot instances (unlike batch clusters that spin up, process, and terminate).

Scale Context:

Uber processes real-time ride events through Apache Kafka for surge pricing, ETA calculation, and fraud detection. The streaming layer handles millions of events per second across thousands of Kafka partitions.
Netflix uses Kafka and Flink for real-time personalization signals: what you are watching right now informs the recommendations shown to other users within seconds.
LinkedIn processes over 7 trillion messages per day through its Kafka infrastructure, powering real-time activity feeds, notifications, and monitoring.
Stripe uses stream processing for real-time fraud scoring on every payment attempt, with a latency budget of under 100 milliseconds.

Interview Tip:

When an interviewer says “real-time,” clarify what they mean. “Real-time” can mean sub-second (fraud detection), sub-minute (live dashboard), or “within a few minutes” (near-real-time analytics). The latency requirement determines whether you need true stream processing (Flink, Kafka Streams), micro-batch (Spark Structured Streaming), or just a frequent batch job. Asking this clarifying question shows engineering maturity.

SLIDE 39: Apache Kafka: Architecture Fundamentals

Apache Kafka is the backbone of modern streaming architectures. It is not a message queue. It is a distributed commit log: an append-only, ordered, durable sequence of records that multiple consumers can read independently. Understanding this distinction is critical because it explains Kafka’s unique properties: retention, replay, and multiple independent consumers.

Core Concepts:

Topics are named channels for data. A topic is a logical category: user_events, order_created, page_views. Producers write to topics; consumers read from topics. Think of a topic as a named log file that multiple systems can write to and read from simultaneously.

Partitions are the unit of parallelism within a topic. Each topic is divided into one or more partitions. Each partition is an independent, ordered, append-only log stored on a single broker. Partitions allow Kafka to scale horizontally: more partitions means more parallel readers.

Brokers are the servers that store and serve data. A Kafka cluster consists of multiple brokers (typically 3-50+ in production). Each broker stores a subset of partitions. Partitions are replicated across multiple brokers for fault tolerance (replication factor, typically 3).

Producers write records to topics. The producer decides which partition a record goes to, either by key-based hashing (all records with the same key go to the same partition) or round-robin (even distribution with no ordering guarantee).

Consumers read records from topics. Consumers track their position (offset) in each partition and read forward sequentially.

Architecture Diagram:

The Commit Log Model:

Unlike traditional message queues that delete messages after consumption, Kafka retains messages for a configurable period (default 7 days, often set to 30 days or longer in production). This retention model enables three critical capabilities:

Replay. A consumer can rewind to an earlier offset and reprocess events. If a bug is discovered in the stream processor, fix the code, reset the offset, and replay the last 7 days of events. This is not possible with traditional queues that delete messages after delivery.
Multiple independent consumers. Two consumer groups can read the same topic at different speeds, at different offsets, for different purposes. Consumer Group A powers the real-time dashboard. Consumer Group B writes to the data lake. Neither blocks the other. Neither “consumes” the message in a way that prevents the other from reading it.
Decoupling. Producers and consumers are fully decoupled. A producer writes to Kafka without knowing or caring which consumers will read the data. New consumers can be added without modifying producers. This is fundamental to building scalable, evolvable data architectures.

SLIDE 40: Kafka: Partitions, Offsets, and Ordering

Partitions and offsets are the mechanics that control parallelism, ordering, and consumer progress in Kafka. Misunderstanding partitioning is one of the most common sources of bugs in streaming systems.

Offsets:

Each record in a partition is assigned a sequential integer called an offset. The offset is the record’s position within that specific partition. Offsets are unique within a partition but not across partitions.

CODE

Partition 0: [offset 0] [offset 1] [offset 2] [offset 3] [offset 4] ...
Partition 1: [offset 0] [offset 1] [offset 2] [offset 3] ...
Partition 2: [offset 0] [offset 1] [offset 2] ...

A consumer tracks its current offset per partition. “I have processed up to offset 47 in partition 0, offset 31 in partition 1, and offset 52 in partition 2.” When the consumer restarts, it resumes from these committed offsets.

Ordering Guarantee:

Kafka guarantees ordering within a single partition only. Records written to partition 0 are read in the exact order they were written. Records across different partitions have no ordering guarantee relative to each other.

This has a critical design implication: if you need ordered processing for an entity (all events for a specific user must be processed in order), you must ensure all events for that entity go to the same partition.

Partition Key Selection:

The producer assigns a partition using a key. All records with the same key are deterministically routed to the same partition (via hash(key) % num_partitions).

Structure & Concept Flow

# All events for user_123 go to the same partition → ordered per-user processing

producer.send("user_events", key="user_123", value=event_data)

# Round-robin (no key) → events distributed evenly but no per-entity ordering

producer.send("user_events", value=event_data)

Choosing the Right Partition Key:

The partition key determines both ordering and load distribution. A good key ensures even distribution across partitions while maintaining the ordering guarantees your application needs.

A key with very low cardinality (e.g., country, with only ~200 values) creates hot partitions. If 40% of users are in one country, one partition handles 40% of the traffic. A key with very high cardinality (e.g., event_id, unique per event) distributes evenly but provides no meaningful ordering.

The sweet spot is a key that matches your processing entity: user_id when you need per-user ordering, order_id when you need per-order ordering, device_id for IoT per-device ordering. These typically have high cardinality (millions of users) and ensure all events for a single entity are processed in order.

Common Misconception:

“Kafka guarantees message ordering.” This is only true within a partition. If a topic has 12 partitions and events are distributed across them, there is no ordering between events on different partitions. If event A (on partition 3) was produced before event B (on partition 7), there is no guarantee that any consumer will see A before B. If global ordering is required (rare, and a performance bottleneck), the topic must have exactly one partition, which eliminates all parallelism.

SLIDE 41: Kafka: Consumer Groups and Scaling

Consumer groups are Kafka’s mechanism for parallel consumption. Understanding them is essential for reasoning about throughput, scaling, and fault tolerance in streaming architectures.

How Consumer Groups Work:

A consumer group is a set of consumers that cooperate to consume a topic. Kafka assigns each partition to exactly one consumer within a group. No two consumers in the same group ever read the same partition.

CODE

Topic: user_events (6 partitions)

Consumer Group A (3 consumers):
  Consumer A1 reads: Partition 0, Partition 1
  Consumer A2 reads: Partition 2, Partition 3
  Consumer A3 reads: Partition 4, Partition 5

Consumer Group B (2 consumers, independent):
  Consumer B1 reads: Partition 0, Partition 1, Partition 2
  Consumer B2 reads: Partition 3, Partition 4, Partition 5

Consumer Group A and Consumer Group B read the same data independently. Within each group, each partition is assigned to exactly one consumer. This model allows multiple downstream systems (dashboard, data lake, alerting) to consume the same topic without interference.

Scaling Rules:

Adding consumers to a group increases parallelism, up to the number of partitions. Beyond that, additional consumers sit idle.

Structure & Concept Flow

Topic with 12 partitions:

3 consumers → each reads 4 partitions → moderate parallelism

6 consumers → each reads 2 partitions → good parallelism

12 consumers → each reads 1 partition → maximum parallelism

15 consumers → 12 active, 3 idle → wasted resources

This is why the number of partitions must be chosen carefully at topic creation time. Partitions set the upper bound on consumer parallelism. A topic with 4 partitions can never have more than 4 parallel consumers (within a single group), regardless of how many consumer instances you deploy.

Rebalancing:

When a consumer joins or leaves a group (due to scaling, deployment, or failure), Kafka triggers a rebalance. Partitions are redistributed among the remaining consumers. During rebalancing, consumption pauses briefly (typically seconds). Frequent rebalancing is a known operational concern in high-throughput systems.

CODE

Before rebalance (3 consumers):
  Consumer 1: Partitions 0, 1, 2, 3
  Consumer 2: Partitions 4, 5, 6, 7
  Consumer 3: Partitions 8, 9, 10, 11

Consumer 3 crashes. Rebalance triggered.

After rebalance (2 consumers):
  Consumer 1: Partitions 0, 1, 2, 3, 4, 5
  Consumer 2: Partitions 6, 7, 8, 9, 10, 11

Consumer 3’s partitions are redistributed to the surviving consumers. Processing resumes from the last committed offsets for those partitions.

SLIDE 41B: Consumer Group Scaling: Worked Example

Scenario: A real-time analytics pipeline processes clickstream events from an e-commerce website. The Kafka topic clickstream has 12 partitions. Traffic varies significantly between peak (Black Friday) and normal periods.

Normal Traffic: 50,000 events/second

Structure & Concept Flow

Each partition: ~4,200 events/second

Consumer processing capacity: ~8,000 events/second per consumer

3 consumers, each reading 4 partitions:

Per consumer: 4 × 4,200 = 16,800 events/second

Consumer capacity: 8,000 events/second

Problem: each consumer is overloaded (2x capacity)

Consumer lag grows continuously → dashboard falls behind

6 consumers, each reading 2 partitions:

Per consumer: 2 × 4,200 = 8,400 events/second

Consumer capacity: 8,000 events/second

Still slightly over capacity → lag grows slowly

8 consumers:

4 consumers read 2 partitions, 4 consumers read 1 partition

Per consumer: 4,200-8,400 events/second

Comfortable headroom on most consumers

Peak Traffic (Black Friday): 200,000 events/second

CODE

Each partition: ~16,700 events/second

12 consumers (maximum for 12 partitions), each reading 1 partition:
  Per consumer: 16,700 events/second
  Consumer capacity: 8,000 events/second
  Problem: even at maximum parallelism, each consumer is overloaded

Solutions:
  1. Increase partition count (requires topic recreation or expansion)
  2. Increase consumer processing capacity (more CPU/memory per consumer)
  3. Simplify processing logic to increase throughput per consumer
  4. Accept higher latency during peak and catch up afterward

Key Takeaway:

Partition count must be chosen with peak traffic in mind, not average traffic. If you create a topic with 6 partitions and later realize you need 24-way parallelism for peak load, adding partitions to an existing topic changes key-to-partition mapping (records that previously went to partition 2 may now go to partition 15), which breaks any ordering guarantees for in-flight data.

The production best practice is to over-provision partitions relative to your current consumer count. Creating a topic with 24 or 48 partitions costs almost nothing in storage or performance, but gives you room to scale consumers up to 24 or 48 when traffic grows. The overhead of extra partitions is minimal (a few MB of metadata); the cost of under-provisioning is an emergency re-partition under production load.

Interview Tip:

When designing a streaming pipeline, state the partition count explicitly and justify it: “I’d create the topic with 24 partitions. Current traffic is 50K events/second, which 6 consumers can handle. But 24 partitions give us room to scale to 24 consumers for 4x peak traffic without re-partitioning.” This demonstrates capacity planning awareness.

SLIDE 42: Kafka Delivery Guarantees

When a producer sends a message to Kafka and a consumer reads it, both can fail at multiple points. Delivery guarantees define what happens to messages when failures occur. This is one of the most common interview questions about Kafka.

At-Most-Once Delivery:

The producer sends the message and does not wait for acknowledgment. If the send fails (network issue, broker down), the message is lost. The consumer commits its offset before processing the message. If the consumer crashes after committing but before processing, the message is skipped on restart.

CODE

Producer: fire and forget (acks=0)
Consumer: commit offset, then process

Failure mode: message loss
When acceptable: metrics collection, logging where occasional loss is tolerable

At-Least-Once Delivery:

The producer retries until the broker acknowledges. If the broker received the first attempt but the acknowledgment was lost, the producer retries and the message is stored twice. The consumer processes the message, then commits the offset. If the consumer crashes after processing but before committing, it reprocesses the message on restart.

CODE

Producer: retry until acknowledged (acks=all, retries=MAX)
Consumer: process, then commit offset

Failure mode: duplicate messages
When acceptable: most use cases, IF the consumer is idempotent

This is the most common production configuration. Duplicates are handled by making the consumer idempotent (upsert on a unique key, or deduplication based on event ID).

Exactly-Once Delivery:

No messages lost, no duplicates. This requires coordination between the producer and consumer through Kafka’s transactional API.

CODE

Producer: idempotent producer (enable.idempotence=true)
  Kafka assigns each producer a unique ID and sequence number per partition.
  If the producer retries, the broker detects the duplicate by sequence number
  and ignores it.

Consumer: transactional consumer
  The consumer reads, processes, and commits the offset in a single Kafka
  transaction. Either all succeed or all roll back.

Failure mode: none (within Kafka's transaction boundary)
When required: financial systems, exactly-once semantics between Kafka topics

The Practical Reality:

Exactly-once within Kafka (topic-to-topic processing via Kafka Streams or Kafka’s transactional API) is well-supported. But most pipelines write to external systems (databases, caches, search indexes), and Kafka’s transactional guarantee does not extend to external systems. In practice, the standard approach is at-least-once delivery with idempotent consumers.

CODE

# At-least-once + idempotent consumer = effectively exactly-once
def process_event(event):
    db.execute("""
        INSERT INTO processed_events (event_id, user_id, amount, processed_at)
        VALUES (%s,%s,%s, NOW())
        ON CONFLICT (event_id) DO NOTHING
    """, (event.id, event.user_id, event.amount))

If the consumer processes the event and crashes before committing the Kafka offset, it reprocesses the same event on restart. The ON CONFLICT DO NOTHING ensures the duplicate is silently ignored.

Interview Tip:

When asked about delivery guarantees, state the practical answer: “I’d configure at-least-once delivery and make the consumer idempotent. This gives us effective exactly-once semantics without the complexity and performance overhead of Kafka transactions. The consumer uses upsert on event_id, so duplicates are harmless.” Then mention that Kafka Streams supports exactly-once for topic-to-topic processing if the interviewer wants to go deeper.

SLIDE 43: Kafka vs Traditional Message Queues

Kafka and traditional message queues (RabbitMQ, SQS, ActiveMQ) are both used for asynchronous communication, but they are fundamentally different systems designed for different patterns. Conflating them in an interview signals a surface-level understanding.

Dimension	Kafka	RabbitMQ	AWS SQS
Core model	Distributed commit log (append-only, ordered, retained)	Message broker (routes messages, deletes after delivery)	Managed message queue (simple, fully managed)
Message retention	Retained for configurable period (days/weeks) regardless of consumption	Deleted after consumer acknowledges	Retained for up to 14 days, deleted after consumption
Replay capability	Yes. Reset consumer offset to any point in retention window.	No. Once consumed and acknowledged, the message is gone.	Limited. Dead-letter queues capture failed messages only.
Ordering	Ordered within partition	No global ordering (unless single queue with single consumer)	FIFO queues provide ordering per message group ID
Consumer model	Consumer groups: each partition read by one consumer in the group	Competing consumers: messages distributed round-robin to consumers	Competing consumers: each message delivered to one consumer
Multiple consumers	Yes. Multiple consumer groups read the same topic independently.	Requires explicit fan-out (exchanges, topic routing)	Requires SNS fan-out to multiple SQS queues
Throughput	Very high (100K+ messages/second per broker)	Moderate (10K-50K messages/second)	Moderate (high with batching, but per-message latency higher)
Delivery latency	Low (milliseconds)	Very low (sub-millisecond for small messages)	Moderate (some polling delay)
Complexity	High (cluster management, partition planning, offset management)	Medium (exchange/queue topology)	Low (fully managed, no infrastructure)

When to Use Each:

Kafka is the right choice when you need durable event streaming: multiple consumers reading the same data, replay capability for reprocessing, high throughput for continuous data pipelines, and long-term retention. Use Kafka for data engineering pipelines, event sourcing, CDC event distribution, and any scenario where the event log is a source of truth.

RabbitMQ is the right choice for task distribution and application-level messaging: distributing work items to a pool of workers, request-reply patterns between microservices, and scenarios where complex routing logic is needed (topic exchanges, header-based routing). Use RabbitMQ when messages should be processed exactly once and discarded, and when the message volume is moderate.

SQS is the right choice when you want a simple, fully managed queue with no infrastructure management. Use SQS for decoupling microservices in AWS, for dead-letter queue patterns, and when the team does not want to operate a Kafka cluster.

The Key Distinction:

Kafka is a log. RabbitMQ and SQS are queues. A log retains data and allows multiple readers at different positions. A queue delivers messages and removes them. In a data engineering context, you almost always want a log (Kafka) because you need multiple consumers (real-time processing + data lake writer + monitoring), replay for reprocessing, and durable retention as a buffer between pipeline stages.

SLIDE 44: Stream Processing Windowing Patterns

Stream processing operates on an unbounded, continuous flow of events. But most useful computations require aggregation over groups of events: “How many clicks in the last 5 minutes?” “What is the average transaction amount per user session?” Windowing is the mechanism that groups events in a stream into finite sets for aggregation.

Tumbling Window

A tumbling window is a fixed-size, non-overlapping window. Every event belongs to exactly one window. When the window closes, the aggregation is emitted and a new window begins.

Structure & Concept Flow

Window size: 5 minutes

Timeline:

|--- Window 1 ---|--- Window 2 ---|--- Window 3 ---|

00:00 00:05 00:05 00:10 00:10 00:15

Events at 00:01, 00:03, 00:04 → Window 1

Events at 00:06, 00:08 → Window 2

Events at 00:11, 00:14 → Window 3

Use tumbling windows for periodic aggregations: “Count events every 5 minutes,” “Sum revenue every hour,” “Average latency per minute.”

CODE

# Flink: count events per user in 5-minute tumbling windows
stream \
    .key_by("user_id") \
    .window(TumblingEventTimeWindows.of(Time.minutes(5))) \
    .aggregate(CountAggregator())

Sliding Window

A sliding window has a fixed size but overlaps with adjacent windows. It advances by a “slide interval” that is smaller than the window size. A single event can belong to multiple windows.

Structure & Concept Flow

Window size: 5 minutes, slide interval: 1 minute

Timeline:

|--- Window A: 00:00-00:05 ---|

|--- Window B: 00:01-00:06 ---|

|--- Window C: 00:02-00:07 ---|

An event at 00:03 belongs to Windows A, B, and C.

Use sliding windows for rolling aggregations: “Show the number of errors in the last 5 minutes, updated every minute.” This produces a smoother view of trends compared to tumbling windows, which can miss patterns that span window boundaries.

CODE

# Flink: error count in sliding 5-minute windows, updated every 1 minute
stream \
    .key_by("service_name") \
    .window(SlidingEventTimeWindows.of(Time.minutes(5), Time.minutes(1))) \
    .aggregate(CountAggregator())

Session Window

A session window is dynamic in size. It groups events that arrive close together in time, with gaps between sessions defining the window boundaries. There is no fixed window size. Instead, a “gap timeout” defines the maximum idle time between events before a new session begins.

CODE

Gap timeout: 30 minutes

User activity:
  Click at 10:00, Click at 10:05, Click at 10:12, Click at 10:15
  [30 minutes of inactivity]
  Click at 10:52, Click at 10:55

Session 1: 10:00 - 10:15 (4 events, 15-minute session)
Session 2: 10:52 - 10:55 (2 events, 3-minute session)

Use session windows for user session analysis: “Calculate pages per session,” “Measure session duration,” “Compute conversion rate per session.”

CODE

# Flink: sessionize user clickstream with 30-minute gap
stream \
    .key_by("user_id") \
    .window(EventTimeSessionWindows.with_gap(Time.minutes(30))) \
    .aggregate(SessionAggregator())

Choosing the Right Window:

Tumbling windows are the simplest and most common. Start with tumbling unless you have a specific reason for sliding or session. Sliding windows add computational cost (each event is processed in multiple windows) but provide smoother trend monitoring. Session windows are the most complex and are used specifically for user behavior analysis where session boundaries are meaningful.

SLIDE 45: Late-Arriving Data and Watermarks

In a perfect world, events arrive at the stream processor in the order they occurred, with no delay. In reality, events arrive out of order and late. A mobile app might buffer events during a subway ride and send them 10 minutes later. A distributed system might process event B before event A due to network routing differences. This creates a fundamental tension between completeness (waiting for late events) and latency (producing results quickly).

The Problem:

CODE

Event timeline (event time = when the event actually happened):
  Event A: occurred at 10:00:00, arrived at processor at 10:00:01
  Event B: occurred at 10:00:02, arrived at processor at 10:00:03
  Event C: occurred at 10:00:01, arrived at processor at 10:00:15  (14 seconds late!)

Tumbling window: 10:00:00 - 10:00:05

If the processor closes the window at 10:00:05 (wall clock):
  Window contains: Event A and Event B
  Event C is missed (arrived 10 seconds after the window closed)
  Result is incomplete.

If the processor waits until 10:00:30 to close the window:
  Window contains: Event A, Event B, and Event C (complete!)
  But: results are delayed by 25 seconds.

Watermarks:

A watermark is the system’s declaration that “all events with event time up to T have (probably) arrived.” When the watermark passes the end of a window, the window is closed and results are emitted.

CODE

Watermark strategy: current_event_time - max_allowed_lateness

If max_allowed_lateness = 10 seconds:
  When the processor sees an event with event_time = 10:00:15,
  it advances the watermark to 10:00:05.
  This means: "I believe all events up to 10:00:05 have arrived."
  The 10:00:00-10:00:05 window can now close.

Allowed Lateness:

Even after the watermark triggers window closure, some systems allow “late” events to update already-emitted results. Flink calls this “allowed lateness.” Events arriving after the watermark but within the allowed lateness period trigger a re-emission of the updated window result.

Structure & Concept Flow

Watermark closes window at T=10:00:05 → emits result (count = 2)

Event C arrives at 10:00:15 with event_time 10:00:01 → within allowed lateness

→ window result re-emitted (count = 3)

Event D arrives at 10:02:00 with event_time 10:00:03 → beyond allowed lateness

→ dropped or sent to a side output for manual handling

The Trade-Off:

Shorter allowed lateness means lower output latency but more incomplete results. Events that arrive after the window closes are lost. Longer allowed lateness means more complete results but higher output latency and higher resource cost (the system must keep window state in memory longer).

Structure & Concept Flow

Allowed lateness: 0 seconds → fastest output, but any late event is lost

Allowed lateness: 10 seconds → handles most network delays

Allowed lateness: 5 minutes → handles mobile app buffering, offline periods

Allowed lateness: 1 hour → handles extreme delays but output is very late

The right value depends on the use case. Real-time fraud detection tolerates almost no lateness (better to miss a late event than delay a fraud decision). Daily analytics dashboards can afford minutes of lateness because the consumer does not notice a 5-minute delay in a chart that updates every minute.

Interview Tip:

When designing a streaming pipeline, always address late data: “Events arrive out of order due to network delays and client-side buffering. I’d set a watermark with 30 seconds of allowed lateness, which handles most network delays while keeping output latency under a minute. Events that arrive more than 30 seconds late are sent to a dead-letter topic for batch reprocessing.” This shows awareness of a real-world problem that most candidates ignore.

SLIDE 46: Stream Processing Frameworks Comparison

Dimension	Kafka Streams	Apache Flink	Spark Structured Streaming	Apache Beam
Deployment model	Library (runs inside your application, no separate cluster)	Standalone cluster (JobManager + TaskManagers)	Spark cluster (Driver + Executors)	Abstraction layer (runs on Flink, Spark, Dataflow)
Processing model	True event-at-a-time	True event-at-a-time	Micro-batch (processes small batches, default 100ms trigger)	Depends on runner (event-at-a-time on Flink, micro-batch on Spark)
Latency	Milliseconds	Milliseconds	Hundreds of milliseconds to seconds (micro-batch latency floor)	Depends on runner
State management	RocksDB (local, embedded)	Managed state backends (RocksDB, heap) with async checkpointing	In-memory state with WAL for recovery	Depends on runner
Exactly-once support	Yes (within Kafka transactions)	Yes (checkpointing + two-phase commit)	Yes (within micro-batch guarantees)	Depends on runner
Windowing	Tumbling, sliding, session	Tumbling, sliding, session, custom windows	Tumbling, sliding (session with workarounds)	Tumbling, sliding, session, custom
Ecosystem coupling	Tightly coupled to Kafka (input and output must be Kafka)	Framework-agnostic (reads from Kafka, files, databases, custom sources)	Integrated with Spark ecosystem (MLlib, SQL, DataFrames)	Multi-runner portability
Operational complexity	Low (no separate cluster, scales with application instances)	High (dedicated cluster management, checkpointing configuration)	Medium (reuses existing Spark infrastructure)	Medium to high (adds abstraction layer)
Best for	Kafka-to-Kafka processing, microservice event processing, lightweight transformations	Complex event processing, low-latency requirements, large-scale stateful processing	Teams already using Spark for batch, near-real-time (seconds, not milliseconds)	Multi-cloud portability, avoiding vendor lock-in

When to Choose Each:

Kafka Streams when the input and output are both Kafka topics, the processing logic is relatively simple (filtering, enrichment, lightweight aggregations), and you want zero operational overhead for a separate processing cluster. Kafka Streams runs as a library inside your Java/Kotlin application and scales by adding application instances.

Flink when you need true event-at-a-time processing with millisecond latency, complex stateful processing (large state, complex windowing, event-time processing with sophisticated watermarks), or when the stream processing is the core of the system (not a side feature). Flink is the gold standard for demanding stream processing workloads.

Spark Structured Streaming when the team already operates Spark for batch and wants a unified framework. Micro-batch processing adds latency (hundreds of milliseconds minimum) but simplifies the programming model. A good choice when “near-real-time” (seconds) is sufficient and true sub-second latency is not required.

Beam when multi-cloud portability is a hard requirement. Beam provides a unified API that runs on Flink, Spark, or Google Cloud Dataflow. The trade-off is an additional abstraction layer that limits access to runner-specific optimizations.

SLIDE 47: Stateful Stream Processing

Most useful stream processing is stateful. Counting events in a window requires remembering the count. Detecting that a user made 5 purchases in 10 minutes requires remembering the previous purchases. Deduplicating events requires remembering which event IDs have been seen. Without state, a stream processor can only filter and transform individual events independently.

Why State Matters:

CODE

Stateless operations (no memory of previous events):
  - Filter: drop events where status != "active"
  - Map: convert temperature from Celsius to Fahrenheit
  - Enrich: add product name by looking up product_id in a static table

Stateful operations (require memory of previous events):
  - Count: "How many events in the last 5 minutes?" (must remember the count)
  - Aggregation: "Running average of transaction amounts" (must remember sum and count)
  - Join: "Combine click events with purchase events for the same user" (must buffer clicks until matching purchase arrives)
  - Deduplication: "Process each event_id only once" (must remember seen IDs)
  - Pattern detection: "Alert if 3 failed logins within 1 minute" (must remember recent failures)

State Stores:

State is stored locally on the stream processor, not in an external database. This is critical for performance: reading state from an external database on every event would add milliseconds of latency and create a database bottleneck.

In Kafka Streams, state is stored in RocksDB, an embedded key-value store running on the same machine as the processor. State is backed up to a Kafka changelog topic for durability. If the processor crashes, the state is rebuilt from the changelog on restart.

In Flink, state is stored in a configurable state backend (RocksDB for large state, heap for small state). State is periodically checkpointed to durable storage (S3, HDFS). On failure, Flink restores the last checkpoint and replays events from Kafka since that checkpoint.

Checkpointing:

Checkpointing is the mechanism that makes stateful stream processing fault-tolerant. The processor periodically snapshots its state to durable storage. If the processor crashes, it restores the last checkpoint and reprocesses events from that point.

CODE

Checkpoint interval: 60 seconds

Timeline:
  t=0:    Checkpoint 1 saved (state: count=1000, offset=5000)
  t=30:   Processing continues (state: count=1500, offset=7500)
  t=45:   CRASH
  t=46:   Restart from Checkpoint 1 (state: count=1000, offset=5000)
  t=47:   Reprocess events 5001-7500 (rebuilds state to count=1500)
  t=60:   Checkpoint 2 saved (state: count=1500, offset=7500)

Events 5001-7500 are reprocessed, so the consumer must be idempotent
or the checkpoint must guarantee exactly-once semantics.

State Size Considerations:

Unbounded state growth is one of the most dangerous anti-patterns in stream processing. If a deduplication processor remembers every event_id forever, its state grows without bound. After months, the state may exceed available disk or memory, causing the processor to fail.

CODE

Unbounded state (dangerous):
  seen_ids = set()  # Grows forever
  def process(event):
      if event.id not in seen_ids:
          seen_ids.add(event.id)
          emit(event)

Bounded state (correct):
  seen_ids = TTLCache(ttl=24_hours)  # Expires entries after 24 hours
  def process(event):
      if event.id not in seen_ids:
          seen_ids.add(event.id)
          emit(event)

In Flink, state TTLs can be configured declaratively. In Kafka Streams, windowed stores automatically expire old windows. The key principle: every piece of state must have an expiration strategy. If state can grow without bound, the system will eventually fail.

SLIDE 48: Worked Example: Real-Time Fraud Detection Pipeline

This worked example walks through a complete streaming pipeline design, applying the concepts from this section. It demonstrates the kind of end-to-end reasoning that interviewers expect for a real-time processing question.

Requirements:

Payment processing platform handling 10,000 transactions per second
Each transaction must be checked for fraud within 500 milliseconds
Fraud signals: (1) more than 5 transactions from the same card in 5 minutes, (2) transaction amount > 3x the card’s 30-day average, (3) two transactions from locations > 500 km apart within 30 minutes
Fraudulent transactions must be flagged in real-time; the payment gateway holds the transaction until the fraud check returns
False positive rate must be < 1% (blocking legitimate transactions is costly)

Step 1: Size the Problem

CODE

Transaction rate:    10,000/second = 864M/day
Transaction size:    ~500 bytes each (card_id, amount, merchant, location, timestamp)
Throughput:          10,000 × 500 bytes = 5 MB/second
Daily volume:        5 MB × 86,400 = 432 GB/day
Latency budget:      500 ms total (network + processing + response)
Processing budget:   ~200 ms (after network overhead)

Step 2: Architecture

Why Flink over Kafka Streams?

The fraud detection logic requires complex stateful processing: per-card transaction history (windowed state), per-card spending averages (long-lived state), and geographic distance calculations. Flink’s managed state backends, sophisticated windowing, and exactly-once checkpointing make it the stronger choice for this level of stateful complexity. Kafka Streams would work for simpler stateless enrichment but is less suited for managing large per-card state.

Step 3: Kafka Topic Design

Structure & Concept Flow

Topic: transactions

Partitions: 64 (allows up to 64 parallel Flink tasks)

Partition key: card_id (all transactions for a card go to the same partition)

Retention: 7 days (for replay if fraud logic is updated)

Replication factor: 3

Why 64 partitions?

10,000 txn/sec ÷ 64 partitions = ~156 txn/sec per partition

Each Flink task can handle 1,000+ txn/sec → comfortable headroom

Room to scale to 64 parallel tasks for future growth

Step 4: Fraud Detection Logic (Flink)

Rule 1: Velocity Check (5 transactions in 5 minutes)

CODE

# Tumbling window: count transactions per card in 5-minute windows
card_transactions \
    .key_by("card_id") \
    .window(TumblingEventTimeWindows.of(Time.minutes(5))) \
    .aggregate(CountAggregator()) \
    .filter(lambda result: result.count > 5) \
    .map(lambda result: FraudAlert(
        card_id=result.card_id,
        rule="velocity",
        details=f"{result.count} transactions in 5 minutes"
    ))

Rule 2: Amount Anomaly (> 3x 30-day average)

CODE

# Stateful processing: maintain per-card running average
# State: {card_id: (total_amount, transaction_count)} with 30-day TTL
def process_transaction(transaction, state):
    avg = state.total_amount / state.transaction_count
    if transaction.amount > 3 * avg:
        emit(FraudAlert(
            card_id=transaction.card_id,
            rule="amount_anomaly",
            details=f"Amount ${transaction.amount} vs avg ${avg:.2f}"
        ))
    state.total_amount += transaction.amount
    state.transaction_count += 1

Rule 3: Geographic Impossibility (500km apart in 30 minutes)

CODE

# Stateful processing: remember last transaction location per card
# State: {card_id: (last_location, last_timestamp)} with 1-hour TTL
def process_transaction(transaction, state):
    if state.last_location is not None:
        distance = haversine(state.last_location, transaction.location)
        time_diff = transaction.timestamp - state.last_timestamp
        if distance > 500 and time_diff < timedelta(minutes=30):
            emit(FraudAlert(
                card_id=transaction.card_id,
                rule="geo_impossibility",
                details=f"{distance:.0f}km in{time_diff.total_seconds()/60:.0f}min"
            ))
    state.last_location = transaction.location
    state.last_timestamp = transaction.timestamp

Step 5: State Management

CODE

State per card:
  Rule 1 (velocity):     window state, auto-expires with window (5 minutes)
  Rule 2 (amount avg):   running average, TTL = 30 days
  Rule 3 (geo):          last location + timestamp, TTL = 1 hour

Total state estimate:
  100M unique cards × ~200 bytes per card = ~20 GB
  Fits in RocksDB state backend across 64 Flink tasks (~300 MB per task)

Checkpointing:
  Interval: 30 seconds
  Storage: S3
  Recovery time: ~10 seconds (restore checkpoint + replay 30 seconds of Kafka)

Step 6: Monitoring

The system monitors processing latency (P99 must stay under 200ms), fraud alert rate (baseline ~0.1% of transactions, alert if it spikes above 1%), checkpoint duration (should complete in < 5 seconds), and consumer lag (should stay under 1,000 events).

SLIDE 49: Wrong vs Right: Stream Processing Anti-Patterns

Anti-Pattern	What Goes Wrong	Correct Approach
Ignoring late-arriving data	Window closes at exactly the window boundary. Events that arrive even 1 second late are silently dropped. Results are systematically incomplete, especially for mobile and IoT sources.	Configure watermarks with appropriate allowed lateness. Send events that arrive after the allowed lateness to a dead-letter topic for batch reprocessing.
Unbounded state growth	Deduplication set or aggregation state grows forever. After weeks or months, the processor runs out of memory or disk and crashes. Recovery time grows proportionally.	Apply TTLs to all state. Use windowed state stores that automatically expire old windows. Monitor state size as a key operational metric.
Assuming ordering across partitions	Logic assumes events arrive in global order. In reality, Kafka only guarantees ordering within a single partition. Events on different partitions can arrive in any relative order.	Partition by the entity that requires ordering (user_id, card_id). Process events per-partition. If global ordering is truly required (rare), use a single partition (sacrificing parallelism).
Synchronous external calls in the processing path	Calling an external API or database synchronously for every event. At 10,000 events/second, even 10ms per call creates a bottleneck. Processing throughput drops to 100 events/second per thread.	Use async I/O (Flink’s AsyncFunction), or enrich events from local state that is periodically refreshed from the external source.
Using streaming for naturally batch workloads	Building a streaming pipeline for daily reports or weekly aggregations. The streaming infrastructure runs 24/7 for a workload that could run in a 2-hour daily batch job.	Match the processing model to the latency requirement. If the consumer tolerates hours of latency, use batch. Reserve streaming for sub-minute requirements.
No backpressure handling	When the consumer falls behind (consumer lag grows), events pile up in Kafka. If the consumer catches up, it processes a burst of old events, potentially overwhelming downstream systems.	Configure consumer backpressure (Flink handles this automatically). Monitor consumer lag and alert when it grows. Scale consumers or increase partition count before lag becomes critical.

SLIDE 50: Interview Practice and Common Mistakes

Common Interview Mistakes in Stream Processing:

Mistake	Why It’s Wrong	Stronger Approach
“We’ll use Kafka” without architecture details	Naming the tool without explaining partitioning, consumer groups, or delivery guarantees.	“I’d create a Kafka topic with 24 partitions keyed by user_id for per-user ordering. Consumer group A powers the real-time dashboard. Consumer group B writes to the data lake. At-least-once delivery with idempotent consumers.”
Proposing exactly-once without understanding the scope	“We’ll enable exactly-once in Kafka.” Kafka’s exactly-once is limited to topic-to-topic processing. It does not guarantee exactly-once for writes to external databases.	“I’d use at-least-once delivery and make the consumer idempotent via upsert on event_id. This is simpler and more robust than Kafka transactions for writes to external systems.”
No mention of late data or watermarks	Designing a windowed aggregation without discussing what happens when events arrive late.	“Events may arrive out of order due to network delays. I’d set a watermark with 30 seconds of allowed lateness. Events beyond that window go to a dead-letter topic.”
Confusing Kafka with a message queue	Treating Kafka like RabbitMQ: assuming messages are deleted after consumption, not mentioning consumer groups or replay capability.	“Kafka retains events for 7 days, allowing us to replay if we discover a bug. Multiple consumer groups read independently, so adding a new consumer doesn’t affect existing ones.”
No state management discussion	Proposing stateful processing (aggregations, deduplication) without discussing state size, TTLs, or checkpointing.	“Per-user state is approximately 200 bytes. With 10M users, total state is ~2 GB, manageable in RocksDB. State TTL is set to 24 hours to prevent unbounded growth. Checkpointing every 60 seconds to S3.”

Practice Questions:

“Design a real-time alerting system that detects when any microservice’s error rate exceeds 5% in a 1-minute window.” (Key patterns: Kafka for error events, tumbling windows, per-service keying, low allowed lateness)
“An e-commerce platform needs real-time inventory updates. When an order is placed, inventory must be decremented within 2 seconds across all channels (web, mobile, in-store).” (Key patterns: Kafka with order_id key, Kafka Streams or Flink consumer, exactly-once semantics or idempotent consumer, state store for inventory counts)
“You’re seeing duplicate events in your streaming pipeline output. Walk me through how you’d diagnose and fix the issue.” (Key patterns: at-least-once delivery causing duplicates on retry, consumer crash before offset commit, fix with idempotent writes or deduplication state)

SLIDE 51: Section Cheat Sheet

Concepts Covered:

Concept	Key Takeaway
When to use streaming	Sub-second to sub-minute latency, event-driven triggers, continuous unbounded data. Not for daily reports or historical analytics.
Kafka architecture	Distributed commit log. Topics contain partitions. Partitions are ordered, append-only, retained. Not a message queue.
Partitions and ordering	Ordering guaranteed within a partition only. Partition key determines which partition an event goes to. Over-provision partitions for future scaling.
Consumer groups	Each partition consumed by exactly one consumer in the group. Max parallelism = number of partitions. Multiple groups read the same topic independently.
Delivery guarantees	At-most-once (may lose), at-least-once (may duplicate), exactly-once (complex). Default: at-least-once with idempotent consumers.
Kafka vs queues	Kafka retains and allows replay. Queues delete after consumption. Kafka supports multiple independent consumers natively.
Windowing	Tumbling (fixed, non-overlapping), sliding (fixed, overlapping), session (dynamic, activity-based). Start with tumbling unless a specific reason requires otherwise.
Late data and watermarks	Events arrive late and out of order. Watermarks declare “all events up to time T have arrived.” Allowed lateness trades completeness for output latency.
Stateful processing	Aggregations, joins, deduplication all require state. State stored locally (RocksDB). Checkpointed periodically for fault tolerance. Every state must have a TTL.

Common Interview Mistakes:

Mistake	Stronger Approach
“Kafka” without architecture	Specify partition count, partition key, consumer groups, delivery guarantee, and retention period.
No late data handling	State your watermark strategy, allowed lateness, and how events beyond the lateness window are handled.
Unbounded state	Specify TTLs for all state, estimate state size, and describe checkpointing strategy.
Streaming for everything	Justify why streaming is needed based on the latency requirement. Most systems are hybrid.
Exactly-once confusion	Distinguish Kafka-internal exactly-once from end-to-end exactly-once. Default to at-least-once with idempotent consumers for external sinks.

Key Numbers to Remember:

Structure & Concept Flow

Kafka Benchmarks:

Single partition throughput: 10,000+ messages/second

Single broker throughput: 100,000+ messages/second

3-broker cluster: 300,000+ messages/second

Typical production partition count: 12-64 per topic (depends on throughput needs)

Default retention: 7 days (configurable to weeks or indefinitely)

Latency Benchmarks:

Kafka end-to-end (producer → consumer): 2-10 ms

Flink event processing: 1-10 ms

Spark Structured Streaming micro-batch: 100-500 ms

Kafka Streams: 1-10 ms

State Sizing:

Per-entity state (counters, averages): 100-500 bytes per entity

10M entities × 200 bytes: ~2 GB total state

100M entities × 200 bytes: ~20 GB total state

RocksDB can handle: hundreds of GB on SSD

Windowing:

Tumbling window output frequency: once per window (e.g., every 5 minutes)

Sliding window computational cost: window_size / slide_interval × events

(5-min window, 1-min slide = each event processed in 5 windows)

How This Connects to Other Sections:

This Section’s Concept	Connects To
Kafka as ingestion backbone	Section 2 (Pipeline Architecture) introduced Kafka as an ingestion mechanism. This section covered its internal architecture. Section 9 (CDC) covers how Debezium publishes change events to Kafka.
Stream vs batch processing	Section 3 (Batch Processing) covered the batch counterpart. Section 5 (Batch vs Streaming) provides the formal decision framework for choosing between them.
Windowing and aggregation	Section 5 (Batch vs Streaming) discusses how Lambda and Kappa architectures handle the overlap between streaming aggregations and batch aggregations.
Kafka delivery guarantees	Section 2 (Pipeline Architecture) introduced idempotency. The at-least-once + idempotent consumer pattern is the same principle applied to streaming.
Stateful processing and state stores	Section 9 (CDC) covers how CDC events build materialized views, which is a form of stateful stream processing (maintaining a continuously updated view from a stream of changes).
Stream processing for real-time serving	Section 7 (Storage Layer) covers where streaming output is served: low-latency stores (Redis), search indexes (Elasticsearch), or directly to another Kafka topic.
Fraud detection worked example	Section 11 (Full Case Studies) integrates streaming into complete architectures: ride-sharing real-time pricing and e-commerce clickstream sessionization both use the patterns from this section.