Sign in to view content

Sign in to view this lesson and continue learning.

Cumulative DAGs in Production Day 3 Lecture

Week 6: Airflow & Pipeline Specs with Trino
41 mins
Data ModelingETL/ELTAirflowData Warehousing

Description

Cumulative DAGs are a complex yet powerful concept in data engineering, particularly when using Apache Airflow. This lecture explores the debate between functional and efficient data engineering, focusing on cumulative table design and its implications for slowly changing dimensions (SCDs). Participants will learn about the trade-offs involved in maintaining simplicity versus efficiency in data pipelines, as well as techniques for effectively managing large datasets while minimizing duplication.