Purchase Required

You need to purchase this content in order to view it

Apache Spark programming with Databricks Day 1 Lab

Module

Description

In this video, Yared discusses about Adaptive Query Execution (AQE) in Spark, highlighting how it optimizes partitioning and joins under the hood, particularly in Spark 3.x. He explaines the differences between on-heap and off-heap memory management, and how Spark handles memory allocation across various components. He also explores Spark UI to track job execution and performance metrics, emphasizing the importance of understanding narrow versus wide transformations.