Skip to content
Hallway
0:00 / 0:00

#paxos#dynamodb#s3

AWS re:Invent 2014 | (SPOT302) Under the Covers of AWS: Its Core Distributed Systems

Amazon Web Services

849K subscribers • 14 videos on Hallway

YouTube
  • Recommended by 1 person: 1 on Hacker News.
  • Presented by Alan Vermeulen and Swami Sivasubramanian
  • Made by the vendor. The company whose product it covers made or sponsored it.
  • May be out of date. Uploaded 11 years ago, in an area where tools change fast.

Advanced. Deep dive for people who know the field.

10K views11 years ago1 recommendation

Distributed systemsCloud and DevOpsConsensus: Raft and PaxosRedis and key-value storesDatabase internals

AWS engineers explain the core distributed systems primitives, such as consensus and transactional logging, that power services like S3 and DynamoDB.

"AWS and Amazon.com operate some of the world's largest distributed systems infrastructure and applications. In our past 18 years of operating this infrastructure, we have come to realize that building such large distributed systems to meet the durability, reliability, scalability, and performance needs of AWS requires us to build our services using a few common distributed systems primitives. Examples of these primitives include a reliable method to build consensus in a distributed system, reliable and scalable key-value store, infrastructure for a transactional logging system, scalable database query layers using both NoSQL and SQL APIs, and a system for scalable and elastic compute infrastructure. In this session, we discuss some of the solutions that we employ in building these primitives and our lessons in operating these systems. We also cover the history of some of these primitives andndash; DHTs, transactional logging, materialized views and various other deep distributed systems concepts; how their design evolved over time; and how we continue to scale them to AWS. "

Why it's here

1 person recommends this

What people say. Highly recommended for its deep insights into how AWS designs, builds, and operates distributed consensus and storage primitives at scale.

rubiquityon Hacker News • Recommends

On “Databases = Frameworks for Distributed Systems”

…tives that Power Our Platform0. That presentation is full of great insight into how AWS thinks about distributed systems. The other presenter, Swami, is the VP of AWS Databases now but started off getting Amazon's initial Paxos implementation going. I was lucky enough to work on those core building blocks and it was an incredible experience. 0 - link to this video…
View on Hacker News

ignoramouson Hacker News • Mentions

On “I followed my dreams to get demoted to software developer”

Reminds me of a "self-demotion" that worked out well: That of a certain Allan Vermeulen 0 who, by some accounts, was the chief architect of Amazon S3 1, and by all accounts, an engineer extraordinaire. Vermeulen managed the Platform Services group when he joined 2, and later moved to a CTO role with focus on the then very nascent Web Services initiative 3, but eventually "demoted" himself to an "AWS Engineer" role to actually build key technological pieces that would underpin a lot of distributed systems at AWS 4. Not bad at all. 0 link 1 link 2 link 3 link 4 link to this video
View on Hacker News

ignoramouson Hacker News • Mentions

On “Consensus Is Harder Than It Looks (2020)”

You are not alone: link and link to this video Like TFA points out, distributed consensus is punishingly hard. AWS relied on TLA+ to prove consensus in DynamoDB and other systems link Interestingly, Kinesis 0 and SQS (?) avoid consensus for those same reasons. 0 link
View on Hacker News

ignoramouson Hacker News • Mentions

On “A Second Conversation with Werner Vogels”

> A few things around S3 were unique. We launched with a set of ten distributed systems tenets in the press release. The ten tenets were separate from S3, stating that this is how you would want to build distributed systems at scale. We just demonstrated that S3 is a really good example of applying those skills. In my own naive way I believe that AWS, today, probably embodies James Hamilton's vision 0 far more than most of those original 10 tenets. - > One of the keys to the success of S3 was that, at launch, it was as simple as it possibly could be, offering little more than GetObject and PutObject. At the time that was quite controversial, as the offering seemed almost too bare bones...…
View on Hacker News

ignoramouson Hacker News • Mentions

On “Paxos vs. Raft: Have we reached consensus on distributed consensus?”

> Basic Paxos, is well, too basic and people mostly run modifications of this to get higher throughput and better latencies. After those modifications, it does not look very different from Raft. Alan Vermeulen, one of the founding AWS engineers, calls inventing newer solutions to distributed consensus an exercise in re-discovering Paxos. link to this video
View on Hacker News

dtjohnnymonkeyon Hacker News • Mentions

On “Why use Paxos instead of Raft?”

I just watched a talk on this the other day (assuming it's same person since he's saying the same thing) the other day: link to this video
View on Hacker News

throwdbaawayon Hacker News • Mentions

On “Summary of the Amazon DynamoDB Service Disruption in US-East-1 Region”

> But I did not expect something as core as the service managing the leases on the physical EC2 nodes to not have recovery procedure. I guess they don't have a recovery procedure for the "congestive collapse" edge case. I have seen something similar, so I wouldn't be frowning at this. A couple of red flags though: 1. Apparent lack of load-shedding support by this DWFM, such that a server reboot had to be performed. Need to learn from link 2. Having DynamoDB as a dependency of this DWFM service, instead of something more primitive like Chubby. Need to learn more about distributed systems primitives from link to this video
View on Hacker News