Sign in to view content

Sign in to view this lesson and continue learning.

Advanced Spark Optimization Techniques Day 1 Lecture

Week 4: Structured Streaming with Spark & Kafka
19 mins
SQLPythonApache Spark

Description

Advanced Spark optimization techniques are explored, focusing on performance and query optimization strategies within Spark. Participants will learn about the Catalyst Optimizer, Adaptive Query Execution (AQE), and memory management, which are crucial for efficient data processing. The session includes a real-life use case related to airport data, demonstrating how to apply these techniques to optimize queries and manage resources effectively in data engineering projects.