Chapter 4: Batch vs Streaming Decision Framework - Data Engineering Course: Complete Interview Preparation | PlayBook Library

SLIDE 52: Section Intro, Batch vs Streaming Decision Framework

Section 5: Batch vs Streaming Decision Framework

Sections 3 and 4 taught you how batch and stream processing work. This section teaches you when to use each. In interviews, “should this be batch or streaming?” is one of the most common follow-up questions after a candidate proposes a pipeline design. The wrong answer is to pick one and defend it rigidly. The right answer is to decompose the system into components with different latency requirements and match each component to the appropriate processing model.

This section gives you the decision framework, the architectural patterns (Lambda, Kappa, hybrid), and the cost and complexity reasoning to justify your choice in any interview.

Recognition Signal:

When the interviewer asks “Should this be real-time or batch?”, “Why not just stream everything?”, “How would you handle both real-time and historical analytics?”, or “Walk me through the trade-offs of your processing model choice,” this section gives you the framework to answer with explicit reasoning rather than intuition.

What We’ll Cover:

The latency-cost-complexity trade-off triangle
Lambda Architecture: dual-path batch + streaming
Kappa Architecture: single-path streaming with replay
Lambda vs Kappa: when each applies
Hybrid architectures in production
A decision flowchart for choosing the processing model
Micro-batch as a middle ground
Cost comparison at scale

Connection to Previous Sections:

This section synthesizes everything from Sections 3 and 4. Section 3 taught you the tools and patterns for batch (Spark, partitioning, shuffle optimization). Section 4 taught you the tools and patterns for streaming (Kafka, windowing, state management). This section teaches you the judgment to choose between them, which is the actual skill interviewers are testing.

SLIDE 53: The Latency-Cost-Complexity Trade-Off Triangle

Every data processing architecture sits on a triangle with three vertices: latency, cost, and complexity. You can optimize for any two, but the third suffers. Understanding this triangle is the foundation for every batch-vs-streaming decision.

Batch Processing: High Latency, Low Cost, Low Complexity

Batch jobs run on a schedule (hourly, daily). Data is available minutes to hours after the events occur. The infrastructure is simple: a Spark cluster that spins up, processes, and shuts down. Cost is low because you pay only for the compute hours used, and batch clusters can run on spot instances (60-90% cheaper).

Stream Processing: Low Latency, Higher Cost, Higher Complexity

Streaming infrastructure runs 24/7. Data is available within seconds of the events occurring. The infrastructure is more complex: Kafka clusters, stream processing frameworks, state management, checkpointing, and offset management. Cost is higher because the infrastructure cannot be shut down between runs, cannot use spot instances (interruption would disrupt continuous processing), and requires ongoing operational attention.

Micro-Batch: Medium Latency, Medium Cost, Medium Complexity

Micro-batch (Spark Structured Streaming) processes data in small, frequent batches (every 1-5 minutes). It trades some latency for reduced complexity compared to true streaming. The infrastructure is similar to batch but runs more frequently.

Quantified Comparison:

Processing the same workload of 1 million events per hour:

CODE

                    Batch (hourly)     Micro-Batch (5-min)    Streaming (Flink)
Data freshness:     1 hour             5 minutes              seconds
Infra cost:         $200/month         $800/month             $2,000/month
Operational effort: Low (cron + alerts) Medium (more runs)    High (24/7 monitoring)
Failure recovery:   Rerun the job      Rerun micro-batch      Restore checkpoint
State management:   None (recompute)   Minimal                Complex (RocksDB, TTLs)
Scaling model:      Add nodes per job  Add nodes per job      Add partitions + consumers
Spot instance use:  Yes (60-90% off)   Partial                No

These are approximate figures and vary by cloud provider and workload, but the relative differences are consistent. Streaming costs 5-10x more than batch for the same throughput because of the always-on infrastructure and the inability to use spot instances.

The Key Insight:

The question is never “which is better?” The question is “what latency does each component of the system actually need?” Most systems have components with different latency requirements, which leads to hybrid architectures where batch and streaming coexist.

SLIDE 54: Lambda Architecture

Lambda Architecture is a pattern for systems that need both real-time results and guaranteed-correct historical results. It runs two parallel processing paths on the same data and merges their outputs.

The Three Layers:

Batch Layer: Processes all historical data periodically (e.g., daily Spark jobs). Produces complete, correct results with high latency. This is the “source of truth” because it sees the entire dataset and can apply complex corrections, deduplication, and aggregations.

Speed Layer: Processes recent data in real-time (e.g., Flink or Kafka Streams). Produces fast, approximate results that cover only the time since the last batch run. These results are temporary and will be overwritten by the next batch run.

Serving Layer: Merges the batch views and real-time views to present a unified result to consumers. Queries combine the latest batch output (complete through yesterday) with the speed layer output (today’s data) to produce an up-to-date view.

Structure & Concept Flow

┌─────────────────────────────┐

│ Batch Layer │

All data ────────────►│ (Daily Spark job) │──► Batch Views

│ │ Complete, correct, slow │ (yesterday and earlier)

│ └─────────────────────────────┘

│ │

│ ┌─────────────────────────────┐ ▼

└───────────────►│ Speed Layer │ ┌──────────┐

│ (Flink / Kafka Streams) │──► │ Serving │──► Consumer

│ Fast, approximate, recent │ │ Layer │

└─────────────────────────────┘ └──────────┘

(merges both)

Concrete Example: Real-Time Dashboard with Historical Accuracy

An analytics dashboard shows daily active users (DAU) and trending content.

The batch layer runs a daily Spark job at 2 AM that computes exact DAU, deduplicating across all sessions, correcting for late-arriving events, and reconciling with source data. This produces the “official” DAU numbers through yesterday.

The speed layer runs a Flink job that counts unique users in real-time using a HyperLogLog approximation. This produces today’s estimated DAU, which is fast but may be slightly inaccurate (HyperLogLog has ~2% error).

The serving layer presents yesterday’s exact DAU (from batch) and today’s estimated DAU (from streaming). When tomorrow’s batch runs, it replaces today’s estimate with an exact count.

Trade-Offs:

The strength of Lambda is that real-time results are available immediately while batch results guarantee correctness. The weakness is significant: you must write and maintain the same logic in two different systems (Spark for batch, Flink for streaming). These two codebases must produce compatible results, and keeping them in sync is a constant maintenance burden. Bugs can exist in one path but not the other, leading to inconsistencies between real-time and batch results.

SLIDE 55: Kappa Architecture

Kappa Architecture eliminates the dual-codebase problem of Lambda by using a single processing path for both real-time and historical data. Everything flows through the streaming layer. Reprocessing historical data means replaying the event log through updated streaming logic.

How It Works:

All data is treated as a stream of events. The event log (Kafka with long retention) is the source of truth. A single stream processing application (Flink, Kafka Streams) processes events and produces output. When logic changes and historical data needs reprocessing, a new instance of the stream processor reads from the beginning of the event log (or a specific offset), reprocesses everything, and writes to a new output. Once the new output catches up to the present, traffic is switched to the new output, and the old output is discarded.

Structure & Concept Flow

Event Log (Kafka, long retention)

│

▼

Stream Processor (single codebase)

│

▼

Output (database, warehouse, cache)

Reprocessing:

1. Deploy new version of stream processor

2. New processor reads from beginning of event log

3. New processor writes to a new output table/topic

4. Once caught up, switch consumers to new output

5. Delete old output

Concrete Example: Reprocessing After a Bug Fix

A stream processor computes per-user spending totals. A bug is discovered: transactions with a specific discount code were being double-counted for the last 3 months.

In Kappa, the fix is:

Fix the stream processing code
Deploy a new instance reading from 3 months ago in the Kafka log
The new instance reprocesses all events with the corrected logic
Once caught up to present, switch the dashboard to read from the new output
Shut down the old instance and delete the old output

There is no separate batch job. The same streaming code handles both real-time and historical processing.

Requirements for Kappa:

Kappa works when three conditions are met:

First, the event log must be replayable. Kafka must retain events for the full reprocessing window (months or longer). At high volume, this requires significant storage. 1 million events/second at 500 bytes each for 90 days is approximately 3.9 PB of Kafka storage.

Second, the same logic must apply to both real-time and historical data. If batch processing uses different algorithms than streaming (e.g., batch uses exact distinct counts while streaming uses HyperLogLog approximations), Kappa does not apply cleanly.

Third, the streaming framework must handle the throughput of replaying historical data. Reprocessing 3 months of data at 10x normal speed requires a cluster capable of 10x normal throughput during the catch-up phase.

Trade-Offs:

The strength of Kappa is a single codebase, a single processing path, and simpler architecture. The weakness is that reprocessing is tied to the streaming framework’s throughput, event log storage can be expensive at extreme scale, and some computations (complex ML training, large multi-table joins) are genuinely difficult or impractical in a streaming framework.

SLIDE 56: Lambda vs Kappa Decision Framework

Dimension	Lambda	Kappa
Number of codebases	Two (batch + streaming)	One (streaming only)
Maintenance burden	High (keep batch and streaming logic in sync)	Lower (single codebase)
Reprocessing approach	Rerun batch job on stored data	Replay event log through updated stream processor
Correctness guarantee	Batch layer produces exact results	Streaming results are the only results (accuracy depends on streaming logic)
Infrastructure complexity	Higher (Spark cluster + Flink cluster + serving layer merger)	Lower (Kafka + stream processor + output store)
Event log storage	Not required for long retention (batch reads from lake/warehouse)	Required for long retention (must replay from log)
Computation flexibility	Batch layer can run any computation (complex joins, ML training)	Limited to what the streaming framework can handle
When to use	When batch and streaming logic genuinely differ (e.g., exact vs approximate counting), when batch must run computations that streaming cannot handle, or when the organization has strong existing batch infrastructure	When the same logic works for both real-time and historical, when the event log is already the source of truth, and when the team wants to avoid maintaining two codebases

Decision Guide:

Start with Kappa if the same processing logic works for both real-time and historical data, and the event log retention is affordable. Kappa is simpler, has a single codebase, and aligns with modern event-driven architectures.

Choose Lambda if the batch layer performs computations that the streaming layer fundamentally cannot replicate: complex multi-table joins across hundreds of GB, exact computations that require seeing the complete dataset (like precise percentile calculations across all users), or ML model training that requires the entire historical dataset. In these cases, the streaming layer provides fast approximate results while the batch layer provides slow exact results.

The Honest Reality:

In interviews, the strongest answer is often: “I’d start with a Kappa-style architecture for simplicity, with a single stream processing codebase. If we later discover that certain computations (like ML training or complex historical aggregations) don’t fit the streaming model well, I’d add targeted batch jobs for those specific workloads. That gives us a Kappa core with surgical Lambda additions where needed.” This shows pragmatic thinking rather than dogmatic adherence to either pattern.

SLIDE 57: The “It Depends” Reality: Hybrid Architectures

In production, most data platforms at scale are hybrid. They use streaming for components that need low latency and batch for everything else. Neither Lambda nor Kappa in their pure forms describes what most companies actually run. The reality is more pragmatic: each workload gets the processing model that matches its latency and cost requirements.

How Hybrid Works in Practice:

The same raw data (often in Kafka) feeds both streaming and batch paths, but each path serves a different set of consumers with different requirements.

Uber:

Streaming handles real-time surge pricing, ETA calculation, and fraud detection. These need sub-second latency and process individual ride events as they occur.

Batch handles daily driver payments, trip analytics, marketplace health reports, and ML model training for demand forecasting. These need complete daily datasets and tolerance for hours of latency.

Both paths read from the same Kafka topics. The streaming consumers process events immediately. A Kafka-to-S3 connector writes the same events to the data lake for batch processing.

Netflix:

Streaming powers real-time A/B test monitoring, session-level personalization signals, and operational alerting on streaming quality metrics.

Batch powers content recommendation model training (requires the complete viewing history), content performance analytics (daily and weekly reports), and royalty calculations (requires exact accounting, not approximations).

Spotify:

Streaming handles real-time “recently played” and “currently trending” features that update within minutes.

Batch handles Discover Weekly and Daily Mix playlist generation, which run as nightly Spark jobs processing the complete listening history.

Key Insight:

The question is never “batch or streaming for this system.” The question is “which components of this system need streaming, and which are better served by batch?” Decompose the system by consumer and latency requirement, then assign each component independently.

SLIDE 58: Decision Flowchart: Batch, Streaming, or Hybrid

When an interviewer asks you to choose a processing model, work through this decision tree. It is not a rigid algorithm, but it structures your reasoning and ensures you consider the key factors.

Structure & Concept Flow

START: What is the latency requirement?

Is sub-second response required?

→ YES: Streaming (Flink, Kafka Streams)

Examples: fraud detection, real-time bidding, live alerting

→ NO: Continue...

Is sub-minute response required?

→ YES: Streaming or micro-batch

If exact event-time windowing needed → Streaming (Flink)

If approximate timing acceptable → Micro-batch (Spark Structured Streaming)

Examples: live dashboards, real-time inventory, session tracking

→ NO: Continue...

Is the data naturally event-driven?

→ YES but latency tolerance is minutes to hours:

Consider micro-batch or frequent batch (every 5-15 minutes)

Examples: near-real-time analytics, delayed notifications

→ NO: Continue...

Is it a scheduled report or historical aggregation?

→ YES: Batch (Spark, warehouse SQL)

Examples: daily reports, monthly reconciliation, data warehouse refresh

Is it ML model training?

→ YES: Batch (almost always)

ML training requires the complete dataset and benefits from batch

optimization (shuffled reads, GPU scheduling)

Exception: online learning with streaming feature updates

Does the system need BOTH real-time features AND historical analytics?

→ YES: Hybrid

Streaming for real-time components, batch for historical components

Examples: e-commerce (real-time inventory + daily reports),

ride-sharing (real-time pricing + daily payments)

How to Present This in an Interview:

Walk through the flowchart explicitly: “The real-time inventory update needs sub-second latency, so that component uses streaming from Kafka through a Flink consumer into Redis. The daily sales report tolerates 6 hours of latency, so that component uses a daily Spark batch job reading from the data lake into Snowflake. The recommendation model training runs weekly on the complete purchase history, which is a batch Spark job. So the overall architecture is hybrid: streaming for inventory, batch for reports and ML.”

This decomposition-based answer is far stronger than “we’ll use streaming” or “we’ll use batch.” It shows you understand that different components have different requirements.

SLIDE 59: Micro-Batch: The Middle Ground

Micro-batch processing sits between true batch and true streaming. It processes data in small, frequent batches (every 30 seconds to 5 minutes), providing near-real-time latency without the full complexity of event-at-a-time stream processing.

How Micro-Batch Works:

Instead of processing each event individually as it arrives, the system accumulates events for a short interval (the trigger interval), then processes the entire micro-batch at once using the batch processing engine.

Structure & Concept Flow

True streaming (Flink):

event → process → emit → event → process → emit → ...

Latency: milliseconds per event

Micro-batch (Spark Structured Streaming):

[accumulate events for 5 seconds] → process batch → emit

Latency: trigger interval + processing time (seconds to minutes)

Spark Structured Streaming:

Spark Structured Streaming is the most widely used micro-batch framework. Internally, it treats the stream as a series of small DataFrames, each containing the events that arrived during the trigger interval. The same Spark SQL/DataFrame API used for batch works for structured streaming.

CODE

# Spark Structured Streaming: process Kafka events in micro-batches
events = spark.readStream \
    .format("kafka") \
    .option("kafka.bootstrap.servers", "broker:9092") \
    .option("subscribe", "user_events") \
    .load()

parsed = events.select(
    from_json(col("value").cast("string"), schema).alias("data")
).select("data.*")

windowed_counts = parsed \
    .withWatermark("event_time", "30 seconds") \
    .groupBy(window("event_time", "5 minutes"), "event_type") \
    .count()

query = windowed_counts.writeStream \
    .outputMode("update") \
    .format("parquet") \
    .option("path", "s3://lake/streaming_output/") \
    .trigger(processingTime="1 minute") \
    .start()

This job runs a micro-batch every 1 minute. Each micro-batch reads all new events from Kafka since the last batch, applies the windowed aggregation, and writes results.

When Micro-Batch Is the Right Choice:

Micro-batch is ideal when the latency requirement is “minutes, not seconds” and the team already operates Spark infrastructure. Concrete scenarios include near-real-time dashboards that refresh every few minutes, monitoring systems where 1-2 minute delay is acceptable, and pipelines that feed downstream consumers (like search indexes or caches) that tolerate minute-level staleness.

When Micro-Batch Is NOT Sufficient:

Sub-second processing (fraud detection, real-time bidding) cannot use micro-batch because the trigger interval adds a latency floor. Complex event-time windowing with precise watermark handling is less mature in Spark Structured Streaming than in Flink. Session windows require workarounds in Spark Structured Streaming that Flink handles natively.

The Advantage of Micro-Batch:

The same Spark API, the same cluster, and the same developer skills serve both batch and micro-batch workloads. A team running daily Spark batch jobs can add a micro-batch streaming job without learning Flink, managing a Flink cluster, or hiring Flink expertise. This organizational simplicity is often the deciding factor.

SLIDE 60: Cost Comparison: Batch vs Streaming at Scale

This slide puts concrete numbers on the cost difference between batch, micro-batch, and streaming for the same workload. Cost is one of the most under-discussed dimensions in interviews, and raising it proactively signals senior-level thinking.

Workload: 1 Million Events/Hour, 24 Hours/Day

Daily volume: 24M events. At 500 bytes each: 12 GB/day.

Batch (Hourly Spark Job):

CODE

Cluster: 10 executors (m5.xlarge spot instances, $0.05/hour each)
Job duration: ~5 minutes per hourly run
Daily compute: 24 runs × 5 minutes = 2 hours of cluster time
Daily cost: 10 nodes × $0.05/hour × 2 hours = $1.00/day

Monthly compute cost:         ~$30
Monthly Kafka cost (input):   ~$100 (3-broker cluster, shared with streaming)
Monthly S3 storage:           ~$8 (12 GB/day × 30 days × $0.023/GB)
Total monthly:                ~$138
Data freshness:               1 hour (at best)

Micro-Batch (Spark Structured Streaming, 1-minute trigger):

CODE

Cluster: 10 executors (m5.xlarge on-demand, $0.192/hour each)
  Cannot use spot for continuous workload (interruption disrupts processing)
Running: 24/7
Daily compute: 10 nodes × $0.192/hour × 24 hours = $46.08/day

Monthly compute cost:         ~$1,382
Monthly Kafka cost:           ~$100
Monthly S3 storage:           ~$8
Total monthly:                ~$1,490
Data freshness:               1-2 minutes

True Streaming (Flink):

CODE

Cluster: 6 TaskManagers (m5.xlarge on-demand, $0.192/hour each)
  + 1 JobManager (m5.large, $0.096/hour)
Running: 24/7
Daily compute: (6 × $0.192 + 1 × $0.096) × 24 = $29.95/day

Monthly compute cost:         ~$899
Monthly Kafka cost:           ~$100
Monthly state storage (S3):   ~$5
Monthly S3 output storage:    ~$8
Total monthly:                ~$1,012
Data freshness:               seconds

Summary:

CODE

                Batch (hourly)    Micro-Batch (1-min)   Streaming (Flink)
Monthly cost:   ~$138             ~$1,490               ~$1,012
Data freshness: 1 hour            1-2 minutes           seconds
Cost ratio:     1x (baseline)     ~11x                  ~7x

Streaming costs 7-11x more than batch for the same throughput. The cost gap widens at higher volumes because batch can use spot instances and process in concentrated bursts, while streaming and micro-batch infrastructure runs continuously.

When the Cost Is Justified:

The 7-11x premium is justified when the business value of low-latency data exceeds the infrastructure cost. Fraud detection preventing $50K/day in fraudulent transactions easily justifies$ 1,000/month in streaming infrastructure. A live dashboard that no one checks more than twice daily does not justify 10x the cost of an hourly batch job.

Interview Tip:

When proposing a streaming architecture, briefly acknowledge the cost: “The streaming path costs approximately $1,000/month for this throughput, compared to$ 140/month for hourly batch. That premium is justified here because the fraud detection system prevents an estimated $50K/day in losses, so the ROI is clear.” This level of reasoning is rare in interviews and marks you as someone who thinks about systems holistically.

SLIDE 61: Worked Example: Choosing the Right Architecture

Scenario:

An e-commerce platform with 10 million daily active users needs three data capabilities:

Real-time inventory updates: When a customer places an order, inventory counts must update across all sales channels (website, mobile app, in-store POS) within 5 seconds. Overselling (selling an item that is out of stock) costs the company $50 per incident in customer service and fulfillment costs.
Daily sales reports: The finance team needs daily revenue, order counts, and return rates broken down by product category, region, and payment method. Reports must be ready by 7 AM for the morning leadership meeting. A few hours of latency is perfectly acceptable.
Recommendation model refresh: The recommendation engine uses a collaborative filtering model trained on purchase history. The model is retrained on the latest 90 days of data. Hourly refresh is sufficient because user preference signals change slowly.

Analysis:

Component 1: Inventory (Streaming)

Latency requirement is 5 seconds. This is a streaming problem. An order event published to Kafka triggers a Flink consumer that decrements inventory in Redis (the real-time inventory store) and publishes an inventory-updated event for all sales channels. The business cost of $50 per oversell incident at even 100 incidents/day ($ 5,000/day) far exceeds the ~$1,000/month streaming infrastructure cost.

Component 2: Sales Reports (Batch)

Latency requirement is “ready by 7 AM.” A daily Spark job scheduled at 2 AM reads yesterday’s orders from the data lake (Bronze layer), joins with product and customer dimensions, aggregates by the required dimensions, and loads the results into Snowflake. The finance team queries Snowflake via their BI dashboard. No streaming infrastructure needed. A 5-hour processing window for a dataset that fits in a 30-minute Spark job provides ample buffer.

Component 3: Recommendation Model (Batch)

ML model training on 90 days of purchase history is a batch problem by nature. An hourly Spark job computes updated feature vectors from the latest purchases and triggers model retraining. The model is deployed to the feature store. Hourly refresh is sufficient because user preferences change over days, not seconds.

Resulting Architecture:

Structure & Concept Flow

Order Events → Kafka Topic: orders

Streaming Path (inventory):

Kafka → Flink → Redis (inventory) → inventory_updated event → all channels

Latency: < 5 seconds

Cost: ~$1,000/month

Batch Path (daily reports):

Kafka → S3 (Bronze) → Daily Spark Job → Snowflake

Latency: data ready by 5 AM (well within 7 AM SLA)

Cost: ~$150/month

Batch Path (recommendations):

S3 (Bronze, 90 days) → Hourly Spark Job → Feature Store → Model Training

Latency: hourly

Cost: ~$300/month

Total infrastructure: ~$1,450/month

Why Hybrid Is the Right Answer:

Using streaming for all three components would cost approximately $3,000/month and add unnecessary complexity to the reporting and ML pipelines. Using batch for all three components would violate the 5-second inventory requirement, costing the business thousands of dollars daily in oversell incidents. Hybrid gives each component exactly the processing model it needs: streaming where latency matters, batch where it doesn’t.

How to Present This in an Interview:

“I’d decompose the system into three components based on latency requirements. Inventory updates need sub-5-second latency, so that path uses Kafka and Flink into Redis. Daily reports need data by 7 AM, so that’s a 2 AM Spark batch job into Snowflake. Recommendations refresh hourly, which is also a batch Spark job. The architecture is hybrid: streaming for the one component that genuinely needs it, batch for the two components where batch is simpler, cheaper, and sufficient.”

SLIDE 62: Interview Practice and Common Mistakes

Common Interview Mistakes:

Mistake	Why It’s Wrong	Stronger Approach
“Everything should be streaming for the best user experience”	Streaming adds 7-11x infrastructure cost and significant operational complexity. Most data consumers (reports, analytics, ML training) do not need sub-minute latency.	Decompose by latency requirement. Only stream what genuinely needs low latency. State the cost trade-off explicitly.
Choosing Lambda or Kappa dogmatically	Saying “I’d use Lambda” or “I’d use Kappa” without analyzing whether the system’s requirements match the pattern.	Start with Kappa for simplicity (single codebase). Add targeted batch jobs only for workloads that genuinely do not fit streaming (ML training, complex historical joins). Explain why.
No cost analysis	Proposing architecture without discussing infrastructure cost. At scale, the batch-vs-streaming decision can mean the difference between $500/month and$ 5,000/month.	Include at least a rough cost comparison: “Batch costs ~ $150/month for this workload. Streaming costs ~$ 1,000/month. The streaming premium is justified for the fraud detection component because it prevents $50K/day in losses.”
Binary thinking (batch OR streaming)	Treating the entire system as a single monolithic decision.	Decompose into components. Most systems are hybrid. Each component gets the processing model that matches its latency and cost requirements.
“Micro-batch is real-time”	Spark Structured Streaming has a latency floor of hundreds of milliseconds to seconds. It is not suitable for sub-100ms requirements.	Micro-batch provides near-real-time (seconds to minutes). True streaming (Flink, Kafka Streams) provides sub-second latency. Name the distinction and choose based on the specific latency requirement.

Practice Questions:

“A social media platform needs: (a) real-time notification delivery, (b) daily engagement analytics, (c) weekly content moderation reports. Design the processing architecture.” (Expected answer: streaming for notifications, batch for both analytics and moderation reports. Justify why each component maps to its processing model.)
“Your current batch pipeline runs daily and produces analytics that are 24 hours stale. The product team wants ‘real-time analytics.’ How do you approach this?” (Expected answer: clarify what ‘real-time’ means. If 5-minute freshness suffices, micro-batch avoids full streaming complexity. If sub-second is needed, add a streaming layer for the specific metrics that require it, keeping batch for everything else.)
“You’ve designed a Kappa architecture. Six months later, the data science team needs to train models on 2 years of historical data. How do you handle this?” (Expected answer: add a targeted batch path for ML training. Replaying 2 years through the streaming framework is impractical at most scales. This is a natural evolution from Kappa to pragmatic hybrid.)

SLIDE 63: Section Cheat Sheet

Concepts Covered:

Concept	Key Takeaway
Latency-cost-complexity triangle	Batch: high latency, low cost, low complexity. Streaming: low latency, higher cost, higher complexity. You optimize for two vertices at the expense of the third.
Lambda Architecture	Dual-path (batch + streaming) merged in a serving layer. Correct but complex: two codebases to maintain.
Kappa Architecture	Single streaming path with replay for reprocessing. Simpler but requires replayable event log and streaming-capable computation.
Lambda vs Kappa	Start with Kappa for simplicity. Choose Lambda when batch and streaming logic genuinely differ or when computations exceed streaming framework capabilities.
Hybrid architectures	Most production systems are hybrid. Each component gets the processing model that matches its latency requirement.
Decision flowchart	Decompose by component. Sub-second → streaming. Sub-minute → streaming or micro-batch. Minutes to hours → batch. Both real-time and historical → hybrid.
Micro-batch	Middle ground (Spark Structured Streaming). Near-real-time latency (seconds to minutes) with batch-like simplicity. Best when the team already uses Spark.
Cost comparison	Streaming costs 7-11x more than batch for the same throughput due to always-on infrastructure and inability to use spot instances.

Common Interview Mistakes:

Mistake	Stronger Approach
Stream everything	Decompose by latency requirement. Only stream what genuinely needs low latency.
Dogmatic Lambda or Kappa	Start simple (Kappa), add batch selectively where streaming cannot handle the workload.
No cost reasoning	State the cost multiple (streaming is 7-11x) and justify the premium with business value.
Binary batch-or-streaming	Hybrid is the default for complex systems. Each component is evaluated independently.
Micro-batch = real-time	Micro-batch provides minutes, not seconds. Distinguish from true streaming for sub-second needs.

Key Numbers to Remember:

Structure & Concept Flow

Cost Multiples (same throughput):

Batch (hourly): 1x (baseline)

Micro-batch (1-min): ~10x

Streaming: ~7x

Latency Ranges:

Batch: minutes to hours

Micro-batch: seconds to minutes

Streaming: milliseconds to seconds

Common Latency Requirements:

Fraud detection: < 500 ms → streaming

Live dashboard: < 30 seconds → streaming

Near-real-time feed: < 5 minutes → micro-batch or streaming

Daily reports: < 12 hours → batch

ML model training: < 24 hours → batch

Infrastructure Runtime:

Batch cluster: runs during job, shuts down (2-4 hours/day typical)

Streaming cluster: runs 24/7/365

Spot instance use: batch yes, streaming no

How This Connects to Other Sections:

This Section’s Concept	Connects To
Batch processing patterns (Spark, partitioning, shuffle)	Section 3 covered these in detail. This section provides the judgment for when to apply them.
Stream processing patterns (Kafka, Flink, windowing)	Section 4 covered these in detail. This section provides the judgment for when to apply them.
Lambda/Kappa architecture choices	Section 11 (Full Case Studies) applies hybrid architectures to complete systems: ride-sharing uses streaming for pricing and batch for payments, e-commerce uses streaming for inventory and batch for reports.
Cost optimization	Section 12 (Pattern Selection) consolidates cost principles across all processing models. Section 3 (Batch) covered spot instances and cluster right-sizing in detail.
Micro-batch (Spark Structured Streaming)	Section 4 (Stream Processing) compared Spark Structured Streaming with Flink and Kafka Streams. This section positions it as the pragmatic middle ground.