OmarAssadion Hacker News • Recommends
On “Famous outages along with deep postmortems?”
Perhaps not famous, but Bryan Cantrill, who gives my favorite talks, has an interesting and funny talk on one of the Joyent outages: link to this video
0:00 / 0:00
#bryancantrill#joyent#postmortem
Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017
GOTO Conferences
1M subscribers • 88 videos on Hallway • Conference channel
- Recommended by 9 people: 8 on Hacker News and 1 on Lobsters.
- A classic. People kept recommending it from 2017 to 2023.
- Presented by Bryan Cantrill, CTO of Joyent
Intermediate. Assumes working knowledge.
100K views9 years ago9 recommendations
Systems programmingTools and practiceDebuggingSRE and incidents
Bryan Cantrill recounts a major datacenter outage at Joyent to explain the human and technical challenges of debugging systems under pressure.
This presentation was recorded at GOTO Chicago 2017. #GOTOcon #GOTOchgo gotochgo.com Bryan Cantrill - Chief Technology Officer at Joyent @bcantrill ABSTRACT As software is increasingly developed to be deployed as part of a service, the manifestations of defects have changed: the effects of broken software are increasingly unlikely to be felt by merely one user, but many (or even all) -- with concomitant commercial consequences. Debugging service [...] Download slides and read the full abstract here: gotochgo.com/2017/sessions/86 twitter.com/GOTOcon linkedin.com/company/goto- instagram.com/goto_con facebook.com/GOTOConferences #Debugging #DebuggingUnderFire #BryanCantrill CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at gotopia.tech Sign up for updates and specials at gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. youtube.com/user/GotoConferences/?sub_confirmation=1Why it's here
9 people recommend thisWhat people say. Highly recommended as an entertaining and educational talk on incident response, postmortems, and the human factors of system outages.
int0x2eon Hacker News • Recommends
On “WAN router IP address change blamed for global Microsoft 365 outage”
Time to share one of my favorite talks (and speakers) ever - "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" (Bryan Cantrill, GOTO 2017) link to this video
KronisLVon Hacker News • Recommends
On “Ask HN: What is the best postmortem you've seen?”
The best postmortem I've seen was actually a conference talk: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" by Bryan Cantrill (in the GOTO conference, 2017) Here's a YouTube video of it: link to this video Here's the slides: link It goes into detail about a pretty bad outage (when an entire data center was brought down), the human aspects, automation, how they handled it, the various risks, architectures, how things fail and about software development in general.
inactive-useron Lobsters • Recommends
Debugging Under Fire: Keep your Head when Systems have Lost their Mind
event covered
merlincoreyon Hacker News • Recommends
On “Root cause analysis: significantly elevated error rates on 2019‑07‑10”
> In my experience customers deeply detest the idea of waiting around for a failure case to re-occur so that you can understand it better. Bryan Cantrill has a great talk0 about dealing with fires where he says something to the effect of: > Now you will find out if you are more operations or development - developers will want to leave things be to gather data and understand, while operations will want to rollback and fix things as quickly as possible 0 Debugging Under Fire: Keep your Head when Systems have Lost their Mind - Bryan Cantrill: link to this video
pdkl95on Hacker News • Recommends
On “They're deleting my channel, but they don't know why? [video]”
> I run a YouTube channel that is over 10 years old , has over 770,000 subscribers , almost 600 public videos , 120,000,000 views When writing software involved in managing a live, public, massively multi-user system, the traditional unix-style commands that are immediate, often silent, and capable of damaging effects become a really easy way to shoot yourself in the foot. Worse, some commands might let you accidentally shoot everyone's foot on a typo. The traditional example is accidentally typing something like "rm -f * .bak" (note the extra space after the star). For a good discussion of this type of problem, I highly recommend Bryan Cantrill's talk1 about the time an operator accidenta…
KronisLVon Hacker News • Recommends
On “Google admits Kubernetes container tech is too complex”
> Joyent famously took down their whole region by rebooting wrong nodes. In case anyone's interested, here's a pretty funny and educational talk by Bryan Cantrill about that particular incident: GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind link to this video
e12eon Hacker News • Recommends
On “Harbormaster: Anti-Kubernetes for your personal server”
Not (I think) the exacttalk/blog post gp was thinking of - but worth watching IMNHO: "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017" link to this video Ed: oh, here we go I think? > Running Aground: Debugging Docker in Production Bryan Cantrill19,102 views16 Jan 2018 Talk originally given at DockerCon '15, which (despite being a popular presentation and still broadly current) Docker Inc. has elected to delist. link
KronisLVon Hacker News • Recommends
On “Is it time to rewrite the Operating System in Rust? (2018) [video]”
I'll give it a watch and will summarize, because while i'm not always on board with the Rust hype (and therefore want to know more, to help eliminate my biases, one way or the other). That said, i rather enjoyed Bryan Cantrill's talk from 2017, "Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill • GOTO 2017": link to this video So i wouldn't necessarily turn away from any of his videos just because of the sometimes humorous or not awfully serious tone.
wingmanjdon Hacker News • Recommends
On “Hackers claim to have breached Okta systems”
…our own company's RTO is 2w, that sounds like a lot needs to be in place. Part of the business continuity/ disaster recovery is getting management to sign off on those types of numbers, big or small. Make sure they're realistic. You're right that this type of recovery is not fun. Bryan Cantrill gives a great presentation about managing an outage ( link to this video ). One of my biggest takeaways, if you're looking at a sweeping outage and a long haul of a recovery, do sleep management asap with your team. Dead tired people are more likely to make brain dead decisions.…
OmarAssadion Hacker News • Recommends
On “The Story of Mel (1983)”
I’m surprised I haven’t seen this linked yet, but Bryan Cantrill of dtrace, Sun, lawnmower, Joyent, etc, fame gave an amazing talk for Monktoberfest 2016, titled “Oral Tradition in Software Engineering”, which features The Story of Mel 1. Highly recommend checking it out — there are loads of little gems and stories like this throughout. All of his other presentations are great too and definitely worth a listen if you like this sort of thing 2. A couple of my favorites are “Fork Yeah! The Rise and Development of Illumos” 3 and “Debugging Under Fire: Keep your Head when Systems have Lost their Mind” 4. 1: link 2: link 3: link 4: link to this video
cls59on Hacker News • Recommends
On “40 years ago yesterday Air Canada Flight 143 ran out of fuel mid-flight”
Bryan Cantrill draws some excellent parallels between this incident and software systems in his "Debugging Under Fire" talk (beware of semi-automated systems): link to this video
tonyarkleson Hacker News • Mentions
On “The sad state of sysadmin in the age of containers (2015)”
> I'm not sure you do, since you admitted to really being at least part sysadmin, earlier :) Hah! I wish I could remember which talk it was. Bryan Cantrill had a good line about DevOps in one of his talks (I think it was this one: link to this video ). The gist was "you can say you're DevOps, but when the shit hits the fan, you're either going to be Dev or Ops. If you're a Dev, you're going to want to debug the problem before rebooting the failed machine. If you're Ops, you're going to want to reboot the machine as fast as you can to get it back up." Through that lens, I'm definitely pretty far over on the Dev spectrum; when something goes catastrophically wrong and someone reboots a box to "solve the…
znpyon Hacker News • Mentions
On “Google outage – resolved”
It reminds me of this: link to this video -- "GOTO 2017 • Debugging Under Fire: Keep your Head when Systems have Lost their Mind • Bryan Cantrill"
KronisLVon Hacker News • Mentions
On “Upptime/upptime: Uptime monitor and status page powered by GitHub”
…nvironments all on the same network. Though if this topology is retained at scale, things can get a bit interesting. On a similar note, I recall Bryan Cantrill doing an interesting presentation "Debugging Under Fire: Keep your Head when Systems have Lost their Mind" that talked about restarting their whole data center and the implications of that: link to this video…
vaidhyon Hacker News • Mentions
On “Amazon confirms corporate staff cuts that could hit 10k employees”
If it were so obvious, I am wondering why you need to worry about doing it wrong. You talk about trade-offs as if they are fixed in time. That is not the case at all in anything but toy systems. Maybe you need to find someone who can design a quality micro-service? You have no idea what I do and what my competencies are, yet, you are assuming you know a lot more about system design than I do and I am missing something simple. Here is another take from Bryan Cantrill [video ]. Very likely, he does not know anything about services either, I guess.