"Stop Rate Limiting! Capacity Management Done Right" by Jon Moore

Jon Moore explains why rate limiting is the wrong abstraction for protecting microservices and argues for concurrency-based capacity management using Little's Law.

In an era of cloud computing and microservices, it's imperative to be able to isolate clients from one another in terms of the resources they can use--otherwise, one misconfigured or malicious client can make a service unusable for everyone else. Steve Yegge mentions quotas and throttling as key practices in his notorious "platform rant." If you've ever had one of your internal microservices launch an unintentional denial of service attack against another one, this talk is for you! The most common mechanism available in both open source and commercial API gateways is rate limiting: making sure a particular clients sends no more than a certain number of requests per unit time. As it turns out, this is exactly the wrong abstraction for multiple reasons. In this talk, we'll learn about a particularly powerful concept from queuing theory known as Little's Law and use it to understand where rate limiting can break down and why an alternative--concurrency control--can be a superior way of addressing this problem. This leads to an elegant implementation that's still easy for clients to understand. Next, we'll talk about an extension to the basic implementation that permits decentralized enforcement and an adaptive, optimistic algorithm that works well with a constantly changing mix of elastic origin capacity, population of clients, and fluctuating usage. A must-have part of your service's self-defense kit! Jon Moore COMCAST Jon Moore is the Chief Software Architect at Comcast Cable, where he focuses on delivering a core set of scalable, performant, robust software components for the company's varied software product development groups. He specializes in the "art of the possible," finding ways to coordinate working solutions for complex problems and deliver them on time (even in large enterprises). Jon is equally comfortable leading and managing teams and personally writing production-ready code. Jon has a passion for software engineering, continuously learning and then teaching colleagues new ways to deliver working, maintainable software with ever-higher quality and ever-shorter delivery times. His current interests include distributed systems, fault tolerance, building healthy and engaging engineering cultures, and Texas Hold'em. Jon received his Ph.D. in Computer and Information Science from the University of Pennsylvania and currently resides in West Philadelphia, although he was neither born there nor raised there and does not spend most of his days on playgrounds.