Skip to content
0:00 / 0:00

#dtrace#harddrive#latency

Shouting in the Datacenter

Bryan Cantrill

9.1K subscribers • 4 videos on Hallway

YouTube
  • Recommended by 6 people: 4 on Hacker News and 2 on Lobsters.
  • A classic. People kept recommending it from 2011 to 2022.
  • Presented by Bryan Cantrill and Brendan Gregg

Intermediate. Assumes working knowledge.

5.6M views17 years ago6 recommendations

Systems programmingHardwarePerformance and optimizationDebugging

Brendan Gregg and Bryan Cantrill demonstrate how acoustic vibration from shouting at a disk array causes immediate spikes in hard drive latency.

Brendan Gregg from Sun's Fishworks team makes an interesting discovery about inducing disk latency. For a ca. 2020 retrospective on this 2008 video: youtube.com/watch?v=_IYzD_NR0W4#t=28m47s

Why it's here

6 people recommend this

What people say. Recommended as a classic demonstration of how physical vibration and sound can measurably degrade hard drive performance.

itamarston Lobsters • Recommends

Shouting at hard drives in the datacenter (2008)

cnston Lobsters • Recommends

Brendan Gregg Shouting in the Datacenter (2008)

dorianjon Hacker News • Recommends

On “SeatGeek Blog: What An Earthquake Does To Page Load Time”

Also interesting: a Sun engineer uses dtrace to show HDD latency spiking after he yells at his disks link to this video

teraflopon Hacker News • Recommends

On “Acoustic Denial of Service Attacks on HDDs”

The paper links to this entertaining YouTube video from 2009, in which someone demonstrates the effect by literally shouting at a rack of servers: link to this video (EDIT: replaced the link I found via YouTube's search with the original, thanks robotmlg!)

geerlingguyon Hacker News • Recommends

On “Raspberry Pi 2.5 Gbps OMV NAS Performance”

One of the little gems I found while I was putting together the video for this post was "Shouting in the datacenter", an older clip (from the pre-HD era) demonstrating the consequence of doing exactly what the title says: link to this video Warning: loud!

kuonon Hacker News • Recommends

On “Song from 1989 declared a cybersecurity vulnerability for crashing hard drives”

I had the "vibration" problem a few times in me IT career, train, construction work and once elevators. It is the kind of issue that are really hard to diagnose the first time but after the first time it's part of the standard "diagnose suite". I love to show this video about the effect of noise on HDD (be sure to put the volume down) link to this video

donavanmon Hacker News • Mentions

On “Netflix's Open Connect Appliances”

+1, in general. To adda bit of content: 1) Power Hungry, the total power envelope of a rack position is the limiting issue. You can fit 16-20 disks per RU no problem. The constraint on total density is your 5kVa or 10kVa power budget. Watts matter. 3) sensitive to vibration, be very very quiet link to this video 4) Density, actually 4TB is hitting the sweet spot for $ per byte last time I looked. If you need absolute density look for 5 & 6TB Real Soon Now. If you can tolerate some loss variable density disks around 5TB look quite a bit more cost effective. 5) Read ahead, you only need to read ahead by a couple of chunks. For video ball park it at 2MB. You don't really need lots of RAM, think 32-64GB pe…

threepipeproblmon Hacker News • Mentions

On “A Loud Sound Shut Down a Bank's Data Center for 10 Hours”

This didn't surprise me, only because the video of the guy screaming at the hard drive has been stuck in my head for years -- link to this video

nazri1on Hacker News • Mentions

On “Linux Performance”

One thing that I learned from him and have never forgotten is that you can reduce a harddisk (the spindle kind) performance just by screaming at it link to this video

robotmlgon Hacker News • Mentions

On “Acoustic Denial of Service Attacks on HDDs”

That's a reposted video, the original is from Bryan Cantrill (formerly of Sun, now CTO of Joyent): link to this video

donatjon Hacker News • Mentions

On “Hard disks can be turned into listening devices”

Interesting. I just watched a video yesterday about how you could see screaming on hard drive stats in a data center: link to this video

geoffton Hacker News • Mentions

On “Hard Drive of Hearing: Disks That Eavesdrop with a Synthesized Microphone [pdf]”

There is a bit of precedent in Bryan Cantrill's 2008 video of Brendan Gregg shouting at some disks and inducing latency: link to this video (HN discussion last year: link ) But yeah, a) turning it from simply detecting that loud sound is happening to getting a meaningful audio channel out of the sound and b) treating this as a security concern instead of just a performance one (and flipping the direction of the attack, from sound attacking drive performance to drives attacking sound privacy) is novel, AFAIK.

croteon Hacker News • Mentions

On “Academics steal data from air-gapped systems using PC fan vibrations”

It also works the other way around: sound will make harddrives malfunction. This means that one could theoretically use a harddrive as a microphone! See link to this video for a demo, yelling at harddrives increases their latency.

MaulingMonkeyon Hacker News • Mentions

On “Western Digital is trying to redefine the word “RPM””

You might not notice a drive being a tiny bit louder/hotter, but you might notice if you've got an entire NAS rack full of 'em. Don't shout at your JBODs: link to this video

AceJohnny2on Hacker News • Mentions

On “The code worked differently when the moon was full”

As long as you don't traumatize the hard drives by yelling at them, feat. brendangregg & bcantrill: link to this video

throw0101aon Hacker News • Mentions

On “Bcachefs – A New COW Filesystem”

…ngs ran just fine. While not directly involved with ZFS development (IIRC), Bryan Cantrill was very 'ZFS-adjacent' since he used Solaris-based systems for a lot of his career, and he has several rants about firmware that you can find online. A video that went viral many years ago, with Cantrill and Brendan Gregg, is "Shouting in the Datacenter": * link to this video * link (making of)…