Purchase Required

You need to purchase this content in order to view it

Setting Up CI/CD and Unit Testing in Databricks for Reliable Data Pipelines

Module
46 mins

Description

In this video, Zach walks through the process of setting up Continuous Integration/Continuous Deployment (CI/CD) and unit testing in Databricks, with a particular emphasis on enhancing the reliability of data pipelines. He begins by creating a Git folder for an Airflow DBT project and explore the YAML configuration for GitHub Actions, which will trigger tests upon pull requests. The demonstration includes running unit tests and addressing bad data scenarios, highlighting the critical importance of ensuring accurate data transformations.