Data Engineering at Netflix using Apache Spark and Flink with Joan Goyeau

Joan Goyeau discusses Netflix's data engineering challenges, video streaming infrastructure, and their use of Apache Spark with Scala 3.

This episode features Joan Goyeau, a data engineering expert at Netflix. We discuss the unique challenges of supporting a massive user base across various devices for flawless video streaming. Joan also sheds light on Netflix's innovative use of Apache Spark with Scala 3. Links Spark for Scala: github.com/vincenzobaz/spark-scala3 Joan's Kubernetes client: github.com/joan38/kubernetes-client VMAF - Video Multi-Method Assessment Fusion open source project from Netflix: github.com/Netflix/vmaf Data engineer job post for Netflix: github.com/vincenzobaz/spark-scala3 Netflix US Salaries: levels.fyi/companies/netflix/salaries/software-engineer?coun… Sponsors Check out ScalaJobs ( scalajobs.com/ ), the most effective platform to hire Scala Engineering talent. Content 00:00 Introduction 00:20 Data Engineering at Netflix 03:06 CAP Theorem for Streaming 10:23 Media preprocessing 15:02 Unexpected Results from Data Analytics 17:30 Netflix Interactive Content 19:22 Spark & Flink 22:35 Tech Choices within Netflix 25:22 Custom Spark & Flink at Netflix 28:23 Chaos Engineering 30:08 Spark with Scala 3 38:45 Scala at Netflix 40:51 Mill Build Tool 46:49 Scala CLI 51:30 Hiring Process at Netflix 01:00:27 Compensation at Netflix 01:04:55 Developing an Open-Source Kubernetes Client 01:12:41 Testing and Integration with GitHub CI #scala #dataengineering #apachespark #netflix