Sign in to view content

Sign in to view this lesson and continue learning.

Apache Spark programming with Databricks Day 1 Lab

Week 4: Structured Streaming with Spark & Kafka
PythonApache SparkDatabricks

Description

In this video, Yared discusses about Adaptive Query Execution (AQE) in Spark, highlighting how it optimizes partitioning and joins under the hood, particularly in Spark 3.x. He explaines the differences between on-heap and off-heap memory management, and how Spark handles memory allocation across various components. He also explores Spark UI to track job execution and performance metrics, emphasizing the importance of understanding narrow versus wide transformations.