Sign in to view content

Sign in to view this lesson and continue learning.

Cumulative DAGs in Production Day 3 Lab

Week 6: Airflow & Pipeline Specs with Trino
34 mins
MLOpsData ModelingETL/ELTAirflowApache IcebergKafka

Description

Cumulative table design and concurrent writes are the focus of this lab, which is ideal for data engineers looking to enhance their skills with Airflow and Trino. Participants will learn how to create and manage cumulative DAGs, handle staging and production tables, and address race conditions during concurrent writes. The session also covers practical techniques for data quality checks and utilizing array transformations, making it a valuable resource for those working with data pipelines.