Skip to content
0:00 / 0:00

#crdt#operationaltransformation#automerge

Conflict Resolution for Eventual Consistency • Martin Kleppmann • GOTO 2016

GOTO Conferences

1M subscribers • 87 videos on Hallway • Conference channel

YouTube
  • Recommended by 3 people: 3 on Hacker News.
  • Presented by Martin Kleppmann, Researcher at University of Cambridge

Intermediate. Assumes working knowledge.

46K views9 years ago3 recommendations

Distributed systemsDatabases and dataConcurrency and parallelismConsensus: Raft and PaxosReplication and consistency

Martin Kleppmann explains how collaborative editors and distributed databases handle concurrent updates using operational transformation and CRDTs.

This presentation was recorded at GOTO Berlin 2016 gotober.com Martin Kleppmann - Researcher at University of Cambridge @kleppmann ABSTRACT What do collaborative editors like Google Docs, the calendar app on your phone, and multi-datacenter database clusters have in common? Answer: They all need to cope with network interruptions, and still work offline. They all allow state to be updated concurrently in several different places, and asynchronously propagate changes to other nodes. If data is concurrently changed on different nodes, you get conflicts that need to be resolved. There are different approaches to handling those conflicts: some systems let the user manually resolve them; some systems choose one version as the winner and throw away the other versions; and some systems try to merge concurrent updates automatically. For example, Google Docs uses an algorithm called Operational Transform (OT) to perform this merge, while Riak uses Conflict-Free Replicated Datatypes (CRDTs) to achieve a similar thing. In this talk we will explore these algorithms for automatic merging. They start out quite simple, but as we shall see, they soon become fascinatingly mind-bending once you start trying to do more ambitious things. For example, if you wanted to write your own spreadsheet app or graphics software that allows several users to edit the same document concurrently, how would you go about doing that? [...] Download slides and read the full abstract here: gotocon.com/berlin-2016/presentations/show_talk.jsp?oid=7910 RECOMMENDED BOOKS Martin Kleppmann • Designing Data-Intensive Applications • amzn.to/3mk2Roj Martin Kleppmann • Secret Colors: A Gentle Introduction to Cryptography • amzn.to/3PgAcyj twitter.com/GOTOcon linkedin.com/company/goto- instagram.com/goto_con facebook.com/GOTOConferences #EventualConsistency #MartinKleppmann CHANNEL MEMBERSHIP BONUS Join this channel to get early access to videos & other perks: youtube.com/channel/UCs_tLP3AiwYKwdUHpltJPuA/join Looking for a unique learning experience? Attend the next GOTO conference near you! Get your ticket at gotopia.tech Sign up for updates and specials at gotopia.tech/newsletter SUBSCRIBE TO OUR CHANNEL - new videos posted almost daily. youtube.com/user/GotoConferences/?sub_confirmation=1

Why it's here

3 people recommend this

What people say. Highly recommended as an exceptionally clear and engaging introduction to CRDTs and conflict resolution in distributed systems.

marknadalon Hacker News • Recommends

On “Distributed algorithms in NoSQL databases”

…hich we use at gunDB, so that way you can get the advantages of the more optimized CRDTs but still have the rest of your data glued together. Another set of really interesting CRDTs which they didn't mention, are the ones used for making decentralized versions of Google Doc: - Martin's Kleppmann has an exceptionally great/interesting talk on this: link to this video - And we recently did a "layman" animated explainer of a similar algorithm, that explains it for the non-academically minded, with simple analogies to "distributed systems" that happen in every day life: link I'll definitely be referencing the OP's article in the future though. Even though I'm in this world, I constantly get the vernacula…

marknadalon Hacker News • Recommends

On “Automerge: JSON-like data structure for building collaborative apps”

I've spent almost a decade working on CRDTs, largely from an industry perspective but also some academic side. They're great, and not too hard to understand if you take time to think about them. Here are some animated/interactive cartoon explainers we've made to help teach the concepts: - How a CRDT version of Operational Transformation (Google Docs) works: link - Why CRDTs are better than centralized alternatives like PAXOS/RAFT: link Martin Kleppmann's work (author of Automerge) is outstanding, especially check out: link to this video

lachenmayeron Hacker News • Recommends

On “Applied Monotonicity: A Brief History of CRDTs in Riak”

CRDTs are quite actively discussed in the IPFS community, in particular here: link (lots of great resources if you're interested in CRDTs) Also check out automerge, a CRDT that aims to be as JSON-like as possible: link Martin Kleppmann (one of the creators of Automerge) has given several great in-depth talks about it, eg. link to this video

marknadalon Hacker News • Recommends

On “Applied Monotonicity: A Brief History of CRDTs in Riak”

CRDTs are great, and I've seen a lot of quality in Christopher's work. A few other resources for those interested in CRDT subject: - Our cartoon explainer link - An excellent talk by Martin Kleppmann link to this video - Marc Shapiro's in depth talk link

lewisl9029on Hacker News • Recommends

On “Why CRDT didn't work out as well for collaborative editing xi-editor”

…state would make any sense to the application business logic that needs to consume that state, or to the human that needs to reason about the rendered result. That is a completely different can of worms that we'll still have to tackle to build a usable application. Here's great example to illustrate this from Martin Kleppmann's talk on this topic: link to this video Rest of the talk is also highly recommended for anyone interested in an approachable primer on CRDTs. The trade-offs to CRDTs mentioned by the author in the context of text-editors make sense, but I would be curious to hear from the Xray team on what their current thinking on the topic is, given that they have collaborative editing as an ex…

marknadalon Hacker News • Recommends

On “Show HN: Graphite – A decentralized and encrypted Google Docs alternative”

Cool concept, but note: I just tested this, and it is NOT a google docs alternative - it has no realtime sync (or at least, it wasn't working). I work extensively on decentralized/P2P distributed systems and realtime P2P sync is non-trivial, you need to use a specialized CRDT for it: - Here is an interactive animated explainer on how to do a P2P realtime syncing CRDT: link - Here is an excellent more detailed explanation of how such a CRDT works, by Martin Kleppmann: link to this video

marknadalon Hacker News • Recommends

On “Differential Synchronization (2009)”

Neil Fraser went on to apply this work at Google with Google Docs (Operational Transformation, see other comment here). He was also one of my heroes that inspired my work to build a decentralized (and offline-first) version, that I later applied to graphs with link . Some good follow ups are: - Martin Kleppmann's link to this video - Cartoon explainer link

marknadalon Hacker News • Recommends

On “Resistant protocols: How decentralization evolves”

Yes, we did a whole interactive explainer article on this: link (cartoon version) Although, from a algorithm side, you may enjoy a talk by Martin (a CRDT researcher) for a more thorough technical overview: link to this video Our team has a model for applying end-to-end encryption to this as well. It isn't a high priority for us to do it ourselves though, so if you are interested in getting involved, we can help explain all the algorithms for implementation!

marknadalon Hacker News • Recommends

On “Local-first software: You own your data, in spite of the cloud”

TLDR: Local-first software is powered by CRDTs. If you want to learn more about CRDTs, check out: - link (author's project, legit) - link to this video (deep technical talk) - link (my Cartoon Explainer)

marknadalon Hacker News • Recommends

On “Open source collaborative text editors”

For anyone who wants a quick primer on OT / CRDT text algorithms: link ( Cartoon Explainer! ) For a detailed deep dive, including RGA: link to this video

marknadalon Hacker News • Recommends

On “OT and CRDT trade-offs for Real-Time collaboration”

Started, but never finished. :( @nmaro has some pretty good stuff. The cartoon explainers gives an overview. But you'd probably want to do RGA algorithm or one of the newer ones. Martin Kleppmann has a pretty good talk with bunch of paper references: link to this video

z3t4on Hacker News • Mentions

On “Show HN: Airborn – Private Google Docs Alternative”

link to this video About 25 minutes in. And it is called OT, so I stand corrected. It does require a server though. I was confusing it with CRDT.

marknadalon Hacker News • Mentions

On “PeerPad – A realtime P2P collaborative editing tool powered by IPFS”

Hey, David/others I'm very impressed with your work on Y-JS (the CRDT library powering this). I'm the author of gun, which is currently (as far as I know from stats) the most popular CRDT-based data sync (MIT/ZLIB/Apache2 licensed, 6.6K+ stars, link ). Usually I'm pretty skeptical of most systems, but I'm rather particularly impressed with yours. I also do not think you are getting the recognition you deserve, because the truly novel work here is Y-JS not IPFS (even though is IPFS is great), yet Y-JS is almost just a footnote on the page which is sad. The first couple documents (on architecture and stuff) didn't seem to explore the CRDT that is used. I'll have to follow-through on other su…