Sign in to view content

Sign in to view this lesson and continue learning.

Apache Spark programming with Databricks Day 2 Lab

Week 4: Structured Streaming with Spark & Kafka
47 mins
SQLPythonETL/ELTApache SparkDatabricks

Description

In this video, Yared walks through the key features of Delta Live Tables (DLT) and explains how it simplifies structured streaming using both SQL and Python. He demonstrates the creation of streaming tables and materialized views, emphasizing the importance of using Auto Loader for efficient file ingestion, which can process thousands of files simultaneously. A real-life scenario involving airport operations is shared to illustrate data engineering concepts.