Skip to content
Hallway
0:00 / 0:00

#elixir#1brc#profiling

The One Billion Row Challenge in Elixir: From 12 Minutes to 25 Seconds - Raj Rajhans | Code BEAM

Code Sync

15K subscribers • 68 videos on Hallway

YouTube
  • Recommended by 4 people: 4 in engineering communities.
  • Presented by Raj Rajhans

Intermediate. Assumes working knowledge.

4.3K views1 year ago4 recommendations

Programming languagesSystems programmingPerformance and optimizationConcurrency and parallelismData engineering and warehouses

Raj Rajhans demonstrates how to optimize Elixir code for the One Billion Row Challenge, reducing runtime from twelve minutes to twenty-five seconds.

✨ This talk was recorded at Code BEAM Europe 2024. If you're curious about our upcoming event, check codebebeameurope.com ✨ The talk presents a case study on optimizing Elixir code for demanding tasks. The challenge is to read a file containing one billion lines of weather data and generate a report with minimum, average, and maximum temperatures for each station. Through this talk, the audience will learn how to benchmark and profile Elixir code, how to model problems in order to make the best use of concurrency in Elixir, and implement performance-tuning strategies effectively. The talk unfolds as a story of performance transformation. Starting with code that initially took over 10 minutes to process the data, we’ll see how the runtime went down to just 25 seconds. By the end of this talk, attendees will gain a deeper understanding of performance optimization in Elixir and leave equipped with practical tools and techniques to enhance their own applications. Let's keep in touch! Follow us on: 💥 Twitter: twitter.com/CodeBEAMio 💥 Facebook: facebook.com/CodeSyncGlobal 💥 LinkedIn: linkedin.com/company/code-sync 💥 Mastodon: genserver.social/codesync

Why it's here

4 people recommend this

What people say. Recommended as an excellent case study on profiling, benchmarking, and progressively optimizing concurrent data processing in Elixir.

u/nnomaein r/elixir • Recommends

On “Can u give me a suggestion?”

For the data validation look at this video \The One Billion Row Challenge in Elixir: From 12 Minutes to 25 Seconds\ for a good progressive way to optimise the parsing and validation parts. Then for the insertion read \Import a CSV into Postgres using Elixir\. Since it seems like in your case it's all or nothing whether the data gets inserted that two should have you pretty much covered.
16View on Reddit

u/leftsaidtimin r/elixir • Recommends

On “Elixir has the worst memory performance in concurrency tasks of nearly all programming languages”

Remember because of the shared-nothing approach with actors memory needs to be copied for each process. There is no global shared state. Designing a system to avoid needlessly copying memory is tricky This conference talk on the one billion row challenge in elixir is a great example of the steps one takes to optimize code for this kind of task in elixir. link to this video
6View on Reddit

u/the_jesterin r/elixir • Recommends

On “Place of Elixir in a 1 Billion nested loop iteration test?”

This seems like a more visual and less-useful version of the 1brc - which had some people doing it in Elixer for you to enjoy already.
2View on Reddit

u/MichaelJ1972in r/elixir • Recommends

On “Running Elixir Script?”

link to this video In case you want to measure and improve. For running the coffee look for escriot. It's a Erlang and ethnicity feature for running scripts.
1View on Reddit