Skip to content
Hallway
0:00 / 0:00

#dropbox#sharding#b-tree

How We've Scaled Dropbox

Stanford

2.2M subscribers • 18 videos on Hallway

YouTube
  • Recommended by 6 people: 5 on Hacker News and 1 curated list.
  • A classic. People kept recommending it from 2014 to 2018.
  • Presented by Kevin Modzelewski, Dropbox engineer

Advanced. Deep dive for people who know the field.

372K views14 years ago6 recommendations

Distributed systemsDatabases and dataScaling and large-scale architectureDatabase internalsMySQL

Kevin Modzelewski describes the technical challenges and database schema optimizations Dropbox used to scale its file synchronization infrastructure.

(Feburary 22, 2012) Kevin Modzelewski talks about Dropbox and its History. He describes the technological issues faced by Dropbox and the actions they have to take in order to continuously improve it. Stanford University: stanford.edu/ Stanford School of Engineering: soe.stanford.edu/ Stanford Computer Systems Colloquium: stanford.edu/class/ee380/ Stanford University Channel on YouTube: youtube.com/stanford

Why it's here

6 people recommend this

What people say. Highly recommended as a classic, detailed case study on scaling database indexes and system architecture from zero to millions of users.

fermienricoon Hacker News • Recommends

On “How we rolled out one of the largest Python 3 migrations”

I found the Dropbox lecture 1 at Stanford one of the most riveting things ever. There is just so much technology behind Dropbox, it is staggering. There is a reason why it is so much better than iCloud sync, Google Drive, Box or OneDrive. 1 link to this video
View on Hacker News

awesome-scalabilitycurated list on GitHubRecommends

Under “Talk”

Scaling Dropbox - Kevin Modzelewski, Back-end Engineer at Dropbox
View the list

rawnlqon Hacker News • Recommends

On “Facebook's software architecture”

This is a talk about the evolution of Dropbox's architecture from 2012: link to this video . It is incredibly detailed (down to the exact sql schema they use for file metadata etc).
View on Hacker News

bradgnaron Hacker News • Recommends

On “Ask HN: Best-architected open-source business applications worth studying?”

Applications dont really need to be well architected until they are hitting scale. Then the parts of their system that need to relieve pressure will need to be re-architected. This is almost like a case study and there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution. Example: link to this video If you dont want to do youtube case studies there are also books to read about distributed systems. Also reading about cloud architecture can help.
View on Hacker News

jesseendahlon Hacker News • Recommends

On “Congrats Dropbox”

Dropbox is a lot more than a GUI on top of rsync. Even purely from an engineering standpoint (ignoring product & design) that's incorrect. You might enjoy this talk: link to this video
View on Hacker News

testbon Hacker News • Recommends

On “Ask HN: Most effective path to become a solid systems practitioner?”

imo the best way is to look at what other's have built. Here's some of my favorite talks that go from 0 users to millions. Dropbox - link to this video Instagram - link Slack (bonus. not as applicable but good reminder of why initial architecture does matter) - link
View on Hacker News

LrnByTeachon Hacker News • Mentions

On “Ask HN: Best-architected open-source business applications worth studying?”

> Applications don't really need to be well architected until they are hitting scale. very True, 'a system Well architected' before hitting scale is considered OVER Engineering > Then the parts of their system that need to relieve pressure will need to be re-architected. > This is almost like a case study and there are a lot of good talks on youtube from places like dropbox and facebook that explain the problem and solution. Example: link to this video
View on Hacker News

srcreighon Hacker News • Mentions

On “How database indexing works internally”

…r the same thing. At 48:36 they explain why they changed from PRIMARY KEY (id) to PRIMARY KEY (ns_id, latest, id). ns_id is kinda like user_id. So that change groups journal entries for users together on disk Specifically. PRIMARY KEY (id) orders things by creation date whereas PRIMARY KEY (ns_id, latest, id) orders things by ns_id primarily. 1: link to this video…
View on Hacker News

srcreighon Hacker News • Mentions

On “Ask HN: It's 2023, how do you choose between MySQL and Postgres?”

…able. They initially used autoinc id for primary key in MySQL. This causes everybodys file changes to be mixed together in chronological order on disk in a B-Tree. The first optimization they made was to change the primary key to (ns_id, latest, id) so that each users (ns_id) latest versions would be grouped together on disk. Dropbox scaling talk: link to this video If a dropbox user has 1000 files and you can fit 20 file-version rows on each 8kb disk page (400bytes/row), the difference in performance…
View on Hacker News

u/versacebluesin r/compsci • Mentions

On “Building a Large Scale Architecture”

>the one that can serve thousands of concurrent requests and support large number of services. [video This is a very good talk about scaling a system. The important thing to realize is that the kind of system you are talking about is not built by one person over a weekend. It is a natural evolution that occurs after years of responding to case by case scaling concerns. >Would normal REST work? Like normal java spring applications Most of the time yet, probably it will have some sort of abstraction layer ontop of it for defining the interfaces, but other times: a message queue system based on-top of redis will better (e.x. building a distributed data transformation pipeline). a s…
2View on Reddit