Sign in to view content

Sign in to view this lesson and continue learning.

How to build job orchestration that matters

Module
54 mins

Description

In this lecture, Zach dives into the concept of orchestration and its critical role in data pipelines, emphasizing that orchestration is not just about tools like Airflow, but about the entire process from data collection to ROI. He shares insights on common pitfalls in orchestration, such as bad logging and schema changes, and discussed best practices like the right audit publish pattern. He also highlights the importance of idempotency in pipelines to avoid inconsistencies.