Skip to content
Hallway
0:00 / 0:00

#apachespark#sparkdataframe#sparkrdd

Himanshu Arora & Nitya Nand Yadav - 10 things I wish I'd known before using Spark in production

ScalaUA Conference

1.2K subscribers • 26 videos on Hallway

YouTube
  • Recommended by 1 person: 1 in engineering communities.
  • May be out of date. The tools, versions or practices it shows have likely moved on.

Intermediate. Assumes working knowledge.

750 views7 years ago1 recommendation

Databases and dataSystems programmingScalaData engineering and warehousesPerformance and optimization

Himanshu Arora and Nitya Nand Yadav share Spark optimization techniques, covering data serialization, hardware tuning, parallelism, and GC tuning.

Himanshu Arora & Nitya Nand Yadav at #ScalaUA - 10 things I wish I'd known before using Spark in production You have recently started working on Spark and your jobs are taking forever to finish? This talk is for you! We have compiled many spark best practices, optimization and tweaks that we have applied over the years in production to make our jobs faster and less resource consuming. In this talk we learn about advanced spark tuning, data serialization​ formats, storage formats, hardware tuning, better data locality, control over parallelism and GC tuning etc. We will also discover the appropriate usage of RDD, DataFrame and Dataset in order to take full advantage of spark internal optimisations. Follow ScalaUA Conference on Twitter: twitter.com/@ScalaUA_Conf

Why it's here

1 person recommends this

What people say. Listed as a recommended talk from the ScalaUA conference.

u/know_not_muchin r/scala • Recommends

All Videos of ScalaUA conference are now available! All the videos of ScalaUA conference can be seen at link to this video and https://old.reddit.com/r/ScalaConferenceVideos/ Since this conference has exceptionally good topic selection, I am listing down the urls to all the talks below Thinking Functionally - John A. De Goes Big Step to Functional Programming - Alex Zvolinskiy Scheduling and retrying effects with cats - Miklós Martin Conquering Concurrency with Functional Programming - Jakub Kozłowski Scala Schemas with Shapeless - Kévin Rauscher Backpressure over the wire - when things go wrong - Zahari Dichev Things I wish I knew when I started with Akka - Markus Jura [Tou…
69View on Reddit