Sign in to view content

Sign in to view this lesson and continue learning.

Deep Dive On Workflows Day 3 Lecture

Week 4: Structured Streaming with Spark & Kafka
11 mins
MLOpsETL/ELTApache SparkAirflowdbtData WarehousingAWSAzureGCP

Description

Explore the intricacies of orchestration in Databricks with a focus on workflows, particularly for structured streaming using Spark and Kafka. This session covers the advantages of Databricks workflows over traditional orchestration tools, emphasizing cost efficiency and unified management. Learn how to implement Delta Live Tables for data quality tracking and monitor task execution, making it ideal for data engineers and data scientists looking to streamline their data processing pipelines.