Scale By The Bay 2018: Yifan Xing, Consensus Algorithms in Distributed Systems

Yifan Xing introduces the core principles of consensus algorithms in distributed systems, bridging academic theory with industry practices like Kafka.

This talk video is a node in devreal.ai, the developer community graph. It connects to: Talk: Consensus Algorithms in Distributed Syst... — devreal.ai/t/181115-consensus-algorithms-distributed-syst-sc… Speaker: Yifan Xing — devreal.ai/n/yifan-xing Company: Apple — devreal.ai/c/apple Event: Scale by the Bay — devreal.ai/e/181114-scale-by-the-bay-conference Explore these connections and related videos, photos, speakers’ blogs and podcasts, and other By the Bay networks. ----- Many messaging systems that are widely used in the industry, e.g., Kafka, use centralized distributed systems services to achieve reliability and consensus between servers. Companies in the industry use the services; however, only a few of them understand the details of the protocols. This talk brings the principles used in academia to the industry by introducing the common distributed systems protocols implemented underneath the popular services. In addition, this talk will compare the differences between how the protocols are used in both academia and the industry. It provides details of how the protocols, specifically Paxos and Raft, work, including how they elect leaders among servers, how they achieve consensus between machines, and how they reliably process and execute client commands. Therefore, it shows how the systems and services, which use the protocols, are enabled to have fault-tolerance, and to achieve confidentiality, integrity, authenticity, availability, etc. From the reliability and security point of view, the talk discusses how the protocols deal with machine failures, including leader failures and replicas failures. It shows the vulnerabilities and potential security issues exist in the protocols. Last but not least, we'll take a look at what we can do to avoid the vulnerabilities when applying the academic theories in the industry.