Frameless: A More Well-Typed Interface for Spark – Long Cao

Long Cao introduces Frameless, a Typelevel library that provides a more type-safe Dataset API for Apache Spark.

With Spark 2.0, Spark users were introduced to the Dataset API, which sought to combine the static guarantees of types (much like in RDDs) with enhancements from Spark SQL’s Catalyst optimizer, which were previously only available to more a weakly typed DataFrame API. In this introductory level talk, we’ll take a brief look at some of the rough edges encountered when working with Datasets and how Frameless, a Typelevel library attempting to add a more well-typed veneer over Spark, can help. * Material * Slides: slides.com/longcao/intro-to-frameless * Event * Typelevel Summit New York City, March 2017: typelevel.org/event/2017-03-summit-nyc/ * About Speaker * Long Cao is a software engineer focusing on Scala, Spark, and data engineering and has been in New York for the last 5 years. He cares deeply about showing newcomers the benefits of Scala and functional programming. On his off time likes to enjoy climbing, Rocket League, music, sports, and coffee. * Licensing * This recording is licensed under CC-BY 4.0 (creativecommons.org/licenses/by/4.0/).