Data Engineering — Zero to Hero

3h 2m total

Move and transform data reliably at scale. Starting from what the data-engineering role actually is, you'll map the modern data stack and the batch-vs-streaming, OLTP-vs-OLAP divides. Then you'll model data for analytics with dimensional star and snowflake schemas, facts and dimensions, and slowly changing dimensions, and choose between a warehouse, a lake, and a lakehouse (with columnar Parquet under the hood). You'll write advanced analytical SQL — window functions, CTEs, grouping sets, and partition-aware performance — then build transformations the modern way: ETL vs ELT, a dbt-style project of models and tests, and idempotent incremental models. From there you scale out with distributed batch processing in Spark (DataFrames, partitions and shuffles, transformations vs actions, lazy evaluation), move to streaming with Kafka (topics, partitions, consumer groups, delivery semantics, event-driven pipelines), orchestrate everything with Airflow (DAGs, scheduling, backfills, idempotency, retries, sensors), and make it trustworthy with data contracts, tests, freshness and volume checks, lineage, and governance. The capstone ties it together into an end-to-end batch + streaming pipeline that is orchestrated and tested. Assumes working SQL and basic Python (see the SQL and Python courses); this one teaches the data-platform layer on top.

Data EngineeringData PipelinesCareer Data

Includes:

Cheat sheet
Progress tracking
Audio narration
Mentor context

Outline

Enroll to unlock

1.
What Data Engineering Is
22 min3 lessons
1. Locked — enroll to unlock. The role: analysts, engineers, and scientists8 min
2. Locked — enroll to unlock. The modern data stack8 min
3. Locked — enroll to unlock. Two dividing lines: batch vs streaming, OLTP vs OLAP6 min
2.
Data Modeling and Storage
20 min3 lessons
1. Locked — enroll to unlock. Dimensional modeling: the star schema8 min
2. Locked — enroll to unlock. Slowly changing dimensions6 min
3. Locked — enroll to unlock. Warehouse, lake, lakehouse — and Parquet6 min
3.
Analytical SQL at Scale
22 min3 lessons
1. Locked — enroll to unlock. Window functions for analytics8 min
2. Locked — enroll to unlock. CTEs and grouping sets8 min
3. Locked — enroll to unlock. Query performance and incremental patterns6 min
4.
ETL vs ELT and Transformation
20 min3 lessons
1. Locked — enroll to unlock. ETL vs ELT6 min
2. Locked — enroll to unlock. dbt-style models and tests8 min
3. Locked — enroll to unlock. Incremental models and idempotent transforms6 min
5.
Batch Processing with Spark
20 min3 lessons
1. Locked — enroll to unlock. Why distributed compute, and the Spark model6 min
2. Locked — enroll to unlock. DataFrames, transformations, and actions8 min
3. Locked — enroll to unlock. Partitions and shuffles6 min
6.
Streaming and Kafka
20 min3 lessons
1. Locked — enroll to unlock. Kafka: the partitioned log6 min
2. Locked — enroll to unlock. Producers, consumers, and consumer groups6 min
3. Locked — enroll to unlock. Delivery semantics and event-driven pipelines8 min
7.
Orchestration with Airflow
20 min3 lessons
1. Locked — enroll to unlock. DAGs, tasks, and dependencies6 min
2. Locked — enroll to unlock. Scheduling, backfills, idempotency, and retries8 min
3. Locked — enroll to unlock. Sensors and waiting for data6 min
8.
Data Quality, Observability, and Governance
20 min3 lessons
1. Locked — enroll to unlock. Data contracts and assertions6 min
2. Locked — enroll to unlock. Observability: freshness, volume, schema, and lineage8 min
3. Locked — enroll to unlock. Governance and PII6 min
9.
Build an End-to-End Pipeline
18 min3 lessons
1. Locked — enroll to unlock. Design the pipeline6 min
2. Locked — enroll to unlock. Build the batch and streaming paths6 min
3. Locked — enroll to unlock. Ship it: quality gates, deliverables, and next steps6 min

Includes:

Cheat sheet
Progress tracking
Audio narration
Mentor context

Outline

Enroll to unlock

1.
What Data Engineering Is
22 min3 lessons
1. Locked — enroll to unlock. The role: analysts, engineers, and scientists8 min
2. Locked — enroll to unlock. The modern data stack8 min
3. Locked — enroll to unlock. Two dividing lines: batch vs streaming, OLTP vs OLAP6 min
2.
Data Modeling and Storage
20 min3 lessons
1. Locked — enroll to unlock. Dimensional modeling: the star schema8 min
2. Locked — enroll to unlock. Slowly changing dimensions6 min
3. Locked — enroll to unlock. Warehouse, lake, lakehouse — and Parquet6 min
3.
Analytical SQL at Scale
22 min3 lessons
1. Locked — enroll to unlock. Window functions for analytics8 min
2. Locked — enroll to unlock. CTEs and grouping sets8 min
3. Locked — enroll to unlock. Query performance and incremental patterns6 min
4.
ETL vs ELT and Transformation
20 min3 lessons
1. Locked — enroll to unlock. ETL vs ELT6 min
2. Locked — enroll to unlock. dbt-style models and tests8 min
3. Locked — enroll to unlock. Incremental models and idempotent transforms6 min
5.
Batch Processing with Spark
20 min3 lessons
1. Locked — enroll to unlock. Why distributed compute, and the Spark model6 min
2. Locked — enroll to unlock. DataFrames, transformations, and actions8 min
3. Locked — enroll to unlock. Partitions and shuffles6 min
6.
Streaming and Kafka
20 min3 lessons
1. Locked — enroll to unlock. Kafka: the partitioned log6 min
2. Locked — enroll to unlock. Producers, consumers, and consumer groups6 min
3. Locked — enroll to unlock. Delivery semantics and event-driven pipelines8 min
7.
Orchestration with Airflow
20 min3 lessons
1. Locked — enroll to unlock. DAGs, tasks, and dependencies6 min
2. Locked — enroll to unlock. Scheduling, backfills, idempotency, and retries8 min
3. Locked — enroll to unlock. Sensors and waiting for data6 min
8.
Data Quality, Observability, and Governance
20 min3 lessons
1. Locked — enroll to unlock. Data contracts and assertions6 min
2. Locked — enroll to unlock. Observability: freshness, volume, schema, and lineage8 min
3. Locked — enroll to unlock. Governance and PII6 min
9.
Build an End-to-End Pipeline
18 min3 lessons
1. Locked — enroll to unlock. Design the pipeline6 min
2. Locked — enroll to unlock. Build the batch and streaming paths6 min
3. Locked — enroll to unlock. Ship it: quality gates, deliverables, and next steps6 min

Outline

What Data Engineering Is

Data Modeling and Storage

Analytical SQL at Scale

ETL vs ELT and Transformation

Batch Processing with Spark

Streaming and Kafka

Orchestration with Airflow

Data Quality, Observability, and Governance

Build an End-to-End Pipeline

Outline

What Data Engineering Is

Data Modeling and Storage

Analytical SQL at Scale

ETL vs ELT and Transformation

Batch Processing with Spark

Streaming and Kafka

Orchestration with Airflow

Data Quality, Observability, and Governance

Build an End-to-End Pipeline