Skip to content
Hallway
0:00 / 0:00

#etl#lyft#maximebeauchemin

Functional Data Engineering - A Set of Best Practices | Lyft

AI Council

49K subscribers • 9 videos on Hallway

YouTube
  • Recommended by 3 people: 3 in engineering communities.
  • Presented by Maxime Beauchemin, Senior Software Engineer at Lyft

Intermediate. Assumes working knowledge.

84K views8 years ago3 recommendations

Databases and dataSoftware designData engineering and warehousesFunctional programmingSoftware architecture

Maxime Beauchemin explains how applying functional programming principles to data engineering can solve common ETL problems and make pipelines more reproducible.

Download slides: datacouncil.ai/talks/functional-data-engineering-a-set-of-be… ABOUT THE TALK: Batch data processing (also known as ETL)  is time-consuming, brittle, and often unrewarding. Not only that, it’s hard to operate, evolve, and troubleshoot. In this talk, we’ll discuss functional programming paradigm and explore how applying it to Data Engineering can bring a lot of clarity to the process. It helps solving some of the inherent problems of ETL, leads to more manageable and maintainable workloads and helps to implement reproducible and scalable practices. It empowers data teams to tackle larger problems and push the boundaries of what’s possible. ABOUT THE SPEAKER: Maxime Beauchemin works as a Senior Software Engineer at Lyft where he develops open source products that reduce friction and help generate insights from data. He is the creator and a lead maintainer of Apache Airflow [incubating], a data pipeline workflow engine; and Apache Superset [incubating], a data visualization platform; and is recognized as a thought leader in the data engineering field. Before Lyft, Maxime worked at Airbnb on the "Analytics & Experimentation Products team". Previously, he worked at Facebook on computation frameworks powering engagement and growth analytics, on clickstream analytics at Yahoo!, and as a data warehouse architect at Ubisoft. ABOUT DATA COUNCIL: Data Council (datacouncil.ai/) is a community and conference series that provides data professionals with the learning and networking opportunities they need to grow their careers. Make sure to subscribe to our channel for more videos, including DC_THURS, our series of live online interviews with leading data professionals from top open source projects and startups. FOLLOW DATA COUNCIL: Twitter: twitter.com/DataCouncilAI LinkedIn: linkedin.com/company/datacouncil-ai - 🎟️ GET YOUR TICKET TO AI COUNCIL 2026 🎟️ Meet the world's top AI infrastructure minds where architects of AI share what works. Three days of high-quality technical talks and meaningful interactions. → aicouncil.com/sf-2026 ⚡ FIND US: X: x.com/AICouncilConf LinkedIn: linkedin.com/company/aicouncilconf/ Website: aicouncil.com/

Why it's here

3 people recommend this

What people say. Recommended as a must-read resource for data engineers, particularly for its alternative approach to handling slowly changing dimensions.

Moments people linked to

u/sbalnojanin r/dataengineering • Recommends

A must-read data engineering collection I just finished writing up a welcome gift for my newsletter, but I wanted to share at least the list of links here. For comments on all the books & articles, don't hesitate to subscribe to https://www.finishslime.com/. *FWIW: I have read all of these, and I did consider all of them very helpful for my data engineering skills! This is not a bogus collection of what others have shared.* # Books * Designing Data-Intensive Applications: The Big Ideas Behind Reliable, Scalable, and Maintainable Systems \- Martin Kleppmann * Fundamentals of Data Engineering \- Reis & Housley * Data Science for Business \- Provost & Fawcett * Big Data: Principles an…
216View on Reddit

u/wtfzamboin r/dataengineering • Recommends

SCD2 surrogate key lookup, is it necessary? Hello, today I was watching this video from Maxime Beauchemin in which he suggests an alternative strategy to handle SCD, different from type 2. The gist of it is to snapshot all the dim table every day and partition by date. In the video he mentions the several complexities about handling SCD2, in specific one concept I was not aware about is "surrogate key lookup". I researched a bit and landed on this example here (scroll until **Customer dimension key lookup**) from AWS, which shows that the synthetic pkey from the dimension table is added to the fact table. Why is this? Can't one just use the dimension_id and the valid date range to joi…
5View on Reddit

u/Aggravating-Party508in r/dataengineering • Recommends

…me. Especially if we treat partition as the Without modern table formats, this would require: * An ELT job to extract daily snapshots and load them into S3 * Loading these snapshots into a data warehouse with a partition date column * Queries that join to the appropriate partition based on the time context (e.g., like the example in this video [video This approach aligns beautifully with functional principles snapshots are immutable, processing is **deterministic**, and pipelines can be **idempotent**. However, it potentially creates significant data duplication, especially for large dimensions that change infrequently. Especially when we treat partitions as the basic building block.…
1View on Reddit