Challenges that everyone struggles while productionizing Apache Spark workloads - Chetan Khatri

Chetan Khatri discusses common challenges and optimization strategies when running Apache Spark workloads in production, covering shuffles, joins, and memory management.

Challenges that everyone struggles while productionizing Apache Spark workloads - Chetan Khatri 1. Primary data structures (RDD, DataSet, Dataframe) 2. Pragmatic explanation - executors, cores, containers, stage, job, a task in Spark. 3. Parallel read from JDBC: Challenges and best practices. 4. Bulk Load API vs JDBC write 5. An optimization strategy for Joins: SortMergeJoin vs BroadcastHashJoin 6. Avoid unnecessary shuffle and use of coalesce, repartition, HashPartitioner with use cases. Impact on cache, Disk I/O, Leaking memory, Internal shuffle, spark executor, spark driver. 7. What to do when spark default sort doesn’t work!, Alternatives. 8. Why dropDuplicates() doesn’t result consistency, What is alternative. 9. Optimize Spark stage generation plan: reduce unnecessary repetitive Actions. 10. Predicate pushdown with partitioning and bucketing. 11. Why not to use Scala Concurrent ‘Future’ explicitly with Spark jobs.