Deep Dive into the Apache Spark Scheduler (Xingbo Jiang)

Xingbo Jiang explains the inner workings of the Apache Spark Scheduler, including DAG construction, task scheduling, and performance tuning.

Xingbo Jiang , a software engineer at Databricks, talks about a core component of data processing platform, scheduler is responsible for schedule tasks on compute units. Built on a Directed Acyclic Graph (DAG) compute model, Spark Scheduler works together with Block Manager and Cluster Backend to efficiently utilize cluster resources for high performance of various workloads. This talk dives into the technical details of the full lifecycle of a typical Spark workload to be scheduled and executed, and also discusses how to tune Spark scheduler for better performance. Learn more here: databricks.com/session/deep-dive-into-the-apache-spark-sched… Article you might like: databricks.com/session/deep-dive-into-spark-sql-with-advance… About: Databricks provides a unified data analytics platform, powered by Apache Spark™, that accelerates innovation by unifying data science, engineering and business. Read more here: databricks.com/product/unified-data-analytics-platform Connect with us: Website: databricks.com Facebook: facebook.com/databricksinc Twitter: twitter.com/databricks LinkedIn: linkedin.com/company/databricks Instagram: instagram.com/databricksinc/ Databricks is proud to announce that Gartner has named us a Leader in both the 2021 Magic Quadrant for Cloud Database Management Systems and the 2021 Magic Quadrant for Data Science and Machine Learning Platforms. Download the reports here. databricks.com/databricks-named-leader-by-gartner