pg_lake: postgres can be your data lake

Marco Slot explains the engineering internals of pg_lake, an extension that enables Postgres to query Iceberg tables using DuckDB.

This is an engineering conversation around pg_lake - a new OSS Postgres extension that lets you query and manage Iceberg tables directly form Postgres. Marco Slot, who has EXTENSIVE experience, shares with us various engineering internals, like: • how pg_lake makes analytics (literally) 100x faster • why Postgres is architecturally terrible at analytical queries (and how vectorized execution fixes this) • how (and why) pg_lake intercepts query plans and delegates parts of the query tree to DuckDB • Marco's hard-won experience through a decade+ career in Postgres • versatility as the real moat of Postgres • the practical differences in engineering b/w OLTP and OLAP • and a lot more -------------------------------------------------------------------- *TIMELINE* 0:02 What is pg_lake? 2:23 Postgres' 100x slower problem and columnar storage experiments they had to make Postgres fast for analytics 6:00 practical examples and internals 16:20 perf internals - vectorized execution & CPU Optimization 23:00 pg_lake architecture (why DuckDB isn't embedded) and the connection-per-process issue 29:16 how pg_lake intercepts the query plan tree and delegates parts to DuckDB 41:09 Iceberg catalogs 48:24 postgres to iceberg ingestion patterns (and pg_incremental) 53:40 Marco's (long) career: early AWS, Citus, Microsoft, Crunchy Data & Snowflake 1:04:20 Marco's observations around the merging between OLTP and OLAP (and the subtle dev differences there) 1:15:30 reverse ETL 1:33:08 Iceberg as the TCP/IP for tables 1:35:00 Marco's thoughts on the "Just Use Postgres" fever ----------------------------------------- *MARCO* You can find Marco on: - LinkedIn: linkedin.com/in/marcoslot/ - X: x.com/marcoslot - GitHub: github.com/marcoslot ----------------------------------------- *pg_lake* You can find the project on GitHub: - github.com/snowflake-labs/pg_lake ----------------------------------------- *TRANSCRIPT* Feed this into your favorite AI for summarization, or to prompt it specific questions: gist.githubusercontent.com/stanislavkozlovski/65c037a8963e49… (or just send Gemini this video link and ask it) ------------------------------------------ *OTHER PLATFORMS* Watch on Spotify here: open.spotify.com/show/4rAFgwNFm9odEaYBoRAtV5 Apple Podcasts: podcasts.apple.com/us/podcast/topicpartition/id1814926834 General RSS: anchor.fm/s/104fd76e0/podcast/rss ----------------------------------------- If you found anything useful from this episode, please consider supporting our growth (so we can continue delivering valuable content). You can do this by simply liking the video. It takes 2 seconds to do, and recording/producing this takes us 8hrs+