colindeanon Lobsters • Recommends
Zebras All the Way Down - Bryan Cantrill, Uptime 2017
0:00 / 0:00
#bryancantrill#storage#firmware
Zebras All the Way Down - Bryan Cantrill, Uptime 2017
Code & Supply
7.2K subscribers • 1 video on Hallway
- Recommended by 7 people: 6 on Hacker News and 1 on Lobsters.
- A classic. People kept recommending it from 2017 to 2025.
- Presented by Bryan Cantrill
Intermediate. Assumes working knowledge.
49K views9 years ago7 recommendations
Systems programmingDatabases and dataDatabase internalsOperating systemsFault tolerance and failure
Bryan Cantrill discusses the engineering challenges of the stateful data path, focusing on storage, firmware, and rare hardware failure modes.
Zebras all the way down: The engineering challenges of the data path Presented by Bryan Cantrill Much attention is rightfully devoted to the development and deployment of stateless services, but these services are not themselves devoid of persistent state; rather, they rely on other services to manage this state for them. This data path, however -- that stack of software that is emphatically not stateless, being responsible for distributed and/or persistent state -- is entirely different in its constraints and failure modes. This software takes years or even decades to get right, can be arduous to upgrade, and -- even in a post-cloud era -- lives and dies by the fickle whims of hardware and firmware. This talk will reflect on two decades of building the data path, from the dawn of storage networking through modern cloud storage services. Presented at Code & Supply's Uptime conference in Pittsburgh, PA. Learn more at codeandsupply.co uptime.eventsWhy it's here
7 people recommend thisWhat people say. Highly recommended by engineers as an entertaining and insightful talk on low-level hardware, firmware, and storage reliability issues.
gt2on Hacker News • Recommends
On “Ask HN: What are some of the best technical talks you've heard?”
Zebras All the Way Down - Bryan Cantrill, Uptime 2017 link to this video Lots of great tangents that add to the subjects. I love talks like this.
ar0bon Hacker News • Recommends
On “iPhones are allergic to helium”
One of my favorite conference talks is semi related to problems like this. link to this video Brian Cantrill - Zebras all the way down
kbensonon Hacker News • Recommends
On “Replacing a silently failing disk in a ZFS pool”
On this topic (and related in many ways) is Bryan Cantrill's talk Zebras All the way Down.1 The gist is that in medicine there's the concept of diseases that are zebras and horses, rare and common, and the saying goes "when you hear hoof beats think horses not Zebras". Unlike medicine, in software most the easy problems have already been abstracted away or made trivial through our software stacks, so often what's left are Zebras. What used to be the exceedingly rare bugs now account for a large proportion of the odd behavior we see in very robust tech stacks, because the easy bugs have been hunted down and eliminated already. 1: link to this video
kbensonon Hacker News • Recommends
On “Five Years of Btrfs”
There's a good Bryan Cantrill talk about that.1 The gist is that eventually, when you throw enough resources at a problem, all that's left are the really uncommon problem and bugs, and this is specifically what you get in the data path (including drive firmware) where things get harder and harder to figure out as the code gets more hidden and obscure. As with all his talks, you can expect it to be quite entertaining as well as informative and historical (if from his POV). 1: link to this video
ignoramouson Hacker News • Recommends
On “Ask HN: What were the things you did that made the biggest impact at your work?”
In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used 0, but even enterprise-grade hardware would at times (at that scale 1) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly 2. Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good 3. 0 link 1 link 2 link 3 link to this video
Bayarton Hacker News • Recommends
On “Installing Debian on Modern Hardware”
>Why does anyone really care about non-free firmware blobs? Aside from the philosophical considerations, stuff breaking in a black-box firmware that you're lead into believing "just works" is particularly aggravating. I'll link Bryan Cantrill's talk on the subject, he's more entertaining than I am. 1 1 link to this video
thomasjudgeon Hacker News • Recommends
On “Synology Lost the Plot with Hard Drive Locking Move”
For an entertaining/terrifying perspective on firmware, obligatory Bryan Cantrill talk "Zebras All the Way Down" link to this video
floatbothon Hacker News • Mentions
On “Ask HN: What’s your favorite talk from 2017?”
On The Turing Completeness of PowerPoint link Compiling C to printable x86, to make an executable research paper link HDR Photography in Microsoft Excel?! link Zebras All the Way Down link to this video Solving Layout Problems with CSS Grid and Friends link Console Security - Switch link Let's move SMM out of firmware and into the kernel link Bringing Linux back to server boot ROMs with NERF and Heads link Sharing the Chisel link (looks like the actual conference video wasn't released)
nordsieckon Hacker News • Mentions
On “Ask HN: What is your favourite tech talk?”
We Really Don't Know How to Compute: Gerry Sussman - link Zebras All the Way Down: Bryan Cantrill - link to this video Jonathan Blow on Deep Work: Jonathan Blow - link Simple Made Easy: Rich Hickey - link Effective Programs - 10 Years of Clojure: Rich Hickey - link The Last Thing D Needs: Scott Meyers - link
cypharon Hacker News • Mentions
On “Why I'm usually unnerved when modern SSDs die on us”
> skates of magnetic readers flying on a cushion of air above a rapidly rotating disc, with the gap separating a few dozen nanometers, often smaller than the process size in the controller's silicon. Complete aside, the fly-height of a magnetic head is actually fractions of a nanometer (i.e. hundreds of picometers). EDIT: I got this from a talk by Bryan Cantrill1. The fly-height is allegedly 0.8 nanometers (800 picometers). 1: link to this video
cypharon Hacker News • Mentions
On “Why I'm usually unnerved when modern SSDs die on us”
(Sheepishly) My only reference is a talk by Bryan Cantrill1 where he states that an exec at a hard-drive manufacturer stated it was ".8 nanometers". I will try to find a better source than that. EDIT: There is a paper from 2016 which did an analysis in the difference in the flying height of the head during different operations, and it was measured in Angstroms2. I couldn't find one that actually gives a precise value of the flying height. There is a 2011 paper which states that some system they were testing allowed for 4-9 nm flying heights3 which is about half-an-order-of-magnitude larger than the claim -- but that's already 7 years old. 1: link to this video 2: link 3: link
cypharon Hacker News • Mentions
On “Why I'm usually unnerved when modern SSDs die on us”
Bryan Cantrill made this claim in a talk recently1, and the fly-height is apparently 800 picometers. 1: link to this video
ShroudedNighton Hacker News • Mentions
On “Investigating Multi-Account IAM Issues in S3 and CloudFront”
It's worth noting that in a computing context the reliability of this heuristic is questionable relative to what it would be in medicine: link to this video Zebras All the Way Down - Bryan Cantrill, Uptime 2017
floatbothon Hacker News • Mentions
On “Samsung Announces Key-Value SSD Prototype”
Well, on the other hand, better to have the ability to deal with it rather than have everything obfuscated in unfixable proprietary firmware… link to this video
tgbugson Hacker News • Mentions
On “ECC matters”
A relevant Bryan Cantrill talk segment on this, which heightens the paranoia around this. Namely, firmware hiding correctable errors and only reporting uncorrectable errors. link to this video