Interactive Library Catalog

Created by Mukul Raina (Senior Software Engineer & Tech Lead at Microsoft, Oxford MSc), this course provides a comprehensive foundation for data engineering interviews covering the entire pipeline lifecycle. It starts with pipeline architecture (ETL vs ELT, CDC, event streams, idempotency, backfill strategies, reliability patterns with DLQs and checkpointing), then dives into batch processing (MapReduce, Spark architecture with Catalyst optimizer, DAG execution, partitioning, shuffle optimization, broadcast joins, file formats like Parquet/ORC/Avro, and Airflow orchestration), and stream processing (Kafka deep dive with topics/partitions/consumer groups, exactly-once semantics, windowing patterns, watermarks, stateful processing with Flink/Spark Streaming). The course also covers the batch vs streaming decision framework (Lambda vs Kappa architectures, micro-batching, latency-cost-complexity triangle) and data modeling patterns (OLTP vs OLAP, normalization/denormalization, Star and Snowflake schemas, fact tables, Slowly Changing Dimensions Type 2, Data Vault, grain design, materialized views)—equipping you with production-ready knowledge and interview-focused practice questions for FAANG-level data engineering roles.