Sign in to view content

Sign in to view this lesson and continue learning.

Cumulative DAGs in Production Day 3 Lab

Description

Cumulative table design and airflow concurrent writes are explored in this lab session, focusing on the creation and management of production and staging tables within Airflow. Participants will learn how to handle data from Kafka, implement data quality checks, and address race conditions in concurrent writes using Trino and Iceberg. This lesson is ideal for data engineers looking to enhance their skills in building robust data pipelines and managing data integrity in production environments.