Skip to content
Hallway
0:00 / 0:00

#blastradius#shufflesharding#cell-basedarchitecture

AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures (ARC338)

Amazon Web Services

849K subscribers • 14 videos on Hallway

YouTube
  • Recommended by 2 people: 2 on Hacker News.
  • Made by the vendor. The company whose product it covers made or sponsored it.
  • May be out of date. Uploaded 7 years ago, in an area where tools change fast.

Intermediate. Assumes working knowledge.

36K views7 years ago2 recommendations

Distributed systemsCloud and DevOpsFault tolerance and failureScaling and large-scale architectureSoftware architecture

An AWS re:Invent talk explaining design techniques like cell-based architecture and shuffle-sharding to limit the blast radius of system failures.

At AWS, we obsess over operational excellence. We have a deep understanding of system availability, informed by over a decade of experience operating the cloud and our roots of operating Amazon.com for nearly a quarter-century. One thing we've learned is that failures come in many forms, some expected, and some unexpected. It's vital to build from the ground up and embrace failure. A core consideration is how to minimize the "blast radius" of any failures. In this talk, we discuss a range of blast radius reduction design techniques that we employ, including cell-based architecture, shuffle-sharding, availability zone independence, and region isolation. We also discuss how blast radius reduction infuses our operational practices.

Why it's here

2 people recommend this

What people say. Engineers frequently recommend this talk for its clear explanation of cell-based architectures and shuffle-sharding in highly resilient distributed systems.

ignoramouson Hacker News • Recommends

On “An introduction to distributed systems (2017)”

…from their EdgeEngineering/NetEng teams (I haven't seen many other service teams in AWS openly share as much about design as them): - link series of articles on Route53's 100% data-plane availability architecture 1. - link colmmacc 2 (seems to be the eng behind AWS HyperPlane 34?) on 10 design patterns for building resilient systems. - link to this video Peter Vosshall (co-creator of Dynamo 5) presenting "cell-based" design in-use at AWS. ---- 0 link 1 link 2 link 3 link 4 link 5 Decandia, G.; Hastorun, D.; Jampani, M.; Kakulapati, G.; Lakshman, A.; Pilchin, A.; Sivasubramanian, S.; Vosshall, P.; Vogels, W. (2007). "Dynamo: Amazon's Highly Available Key-value Store".…
View on Hacker News

ignoramouson Hacker News • Recommends

On “Post-mortem of March 12 Google outage”

…eMachinesOrStorage); the clients are routed to assigned cells by a highly available routing layer (like NetworkTrafficShapers/DNS), which can also help facilitate migration of clients from one/more cell to another. I encourage you to view the re:invent presentation on it 0 if resiliency in distributed systems is something that interests you. 0 link to this video…
View on Hacker News

talawahtechon Hacker News • Recommends

On “Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region”

…un Cloudwatch, Cognito and Customer workloads on the same set of cells. It is also important to note that celluarization only helps in this case if they limit code deployment to a limited number of cells at a time. This YouTube video1 of a re:invent presentation does a great job of explaining it. The cell-based stuff, starts around minute 20. 1. link to this video…
View on Hacker News

ignoramouson Hacker News • Mentions

On “Post-mortem of March 12 Google outage”

…ge here reminds me of issues CloudFront faced wrt configurations in their earlier years. Here's master presenter Harvo Jones going into a bit more detail: link Kind of vindicates AWS' decision (after major outages to DynamoDB and S3) to not just isolate regions or zones but resources within those regions/zones further into what they call "cells" link to this video Tumblr spoke abt "cells" too in the context of achieving isolation at scale, but in a quite different way: link As these known-knowns (and the Google sre-book is teeming with these) get designed around, it'd be interesting to see why/how future outages would happen.…
View on Hacker News

ignoramouson Hacker News • Mentions

On “Ask HN: What were the things you did that made the biggest impact at your work?”

In my time writing databases, I found that not only the mount-flags could result in data-loss despite fsync, given you use fsync as it is supposed to be used 0, but even enterprise-grade hardware would at times (at that scale 1) drop and/or corrupt a few bits here and there. That reminds me of the case where AWS famously revealed that a single NIC in their ginaromous S3 fleet flipped a single bit once in a while and that caused an outage because their gossip-daemon responsible for fleet health-checks failed spectacularly 2. Bryan Cantrill's talk on the realted topic of hardware/firmware bugs is pretty good 3. 0 link 1 link 2 link to this video 3 link
View on Hacker News

ignoramouson Hacker News • Mentions

On “Summary of the Amazon Kinesis Event in the Northern Virginia (US-East-1) Region”

…ont-end fleet. ...moving the front-end server shard-map cache that takes a long time to build, up to an hour sometimes? to a dedicated fleet. ...move a few large AWS services, like CloudWatch, to a separate, partitioned front-end fleet. ...accelerate the cellularization 0 of the front-end fleet to match what we’ve done with the back-end. 0 link to this video and link…
View on Hacker News

708145_on Hacker News • Mentions

On “Cell-Based Architecture”

The concept have been mentioned by Amazon for years. For reference see: "AWS re:Invent 2018: How AWS Minimizes the Blast Radius of Failures" link to this video
View on Hacker News

faizshahon Hacker News • Mentions

On “Fly.io outage, recently deployed apps down, deployments disabled”

AWS invested a ton into limiting the blast radius of failures by isolating AZs, regions and using a cellular (service level isolated shards) architecture. I am surprised these ideas have not propagated to newer companies trying to build clouds: link to this video AWS isn’t perfect but these lessons were learned by fire because these sorts of global outages can seriously harm reputations.
View on Hacker News

ignoramouson Hacker News • Mentions

On “Slack’s Migration to a Cellular Architecture”

ex-AWS here May be marketing but it is an architecture born out of Amazon's (and AWS's) use of AWS: - Reliable scalability: How Amazon.com scales in the cloud , link (2022) - How AWS minimizes the blast radius of failures , link to this video (2018) For massive enterprise products like Slack that need close to 100% uptime across all their services, cells make sense.
View on Hacker News