Skip to content
0:00 / 0:00

#latency#coordinatedomission#hdrhistogram

"How NOT to Measure Latency" by Gil Tene

Strange Loop Conference

88K subscribers • 106 videos on Hallway • Conference channel

YouTube
  • Recommended by 11 people: 9 on Hacker News, 1 on Lobsters and 1 curated list.
  • A classic. People kept recommending it from 2015 to 2025.
  • Presented by Gil Tene, CTO and co-founder of Azul Systems
  • An opinion. It argues one side of a question engineers disagree on.

Intermediate. Assumes working knowledge.

119K views11 years ago11 recommendations

Systems programmingTools and practicePerformance and optimizationTestingObservability and monitoring

Gil Tene explains common pitfalls in measuring latency, why averages are misleading, and how coordinated omission ruins response time data.

Time is Money. Understanding application responsiveness and latency is critical but good characterization of bad data is useless. Gil Tene discusses some common pitfalls encountered in measuring latency and response time behavior. He introduces how simple, open sourced tools can be used to improve and gain higher confidence in both latency measurement and reporting. Gil Tene AZUL SYSTEMS @giltene Gil Tene is CTO and co-founder of Azul Systems. He has been involved with virtual machine and runtime technologies for the past 25 years. His pet focus areas include system responsiveness and latency behavior. Gil is a frequent speaker at technology conferences worldwide, and an official JavaOne Rock Star. He pioneered the Continuously Concurrent Compacting Collector (C4) that powers Azul's continuously reactive Java platforms. In past lives, he also designed and built operating systems, network switches, firewalls, and laser based mosquito interception systems.

Why it's here

11 people recommend this, 1 disputes it

What people say. Engineers point to this talk as a life-changing explanation of why averages fail for latency measurement and how to properly use percentiles.

vikiomega9on Hacker News • Recommends

On “Throughput vs. Latency and Lock-Free vs. Wait-Free”

Doesn't this depend on how "tuned" your subcomponents are? If I had a system that's supposedly heavily optimized, I would simply start seeing the worst-case latency on a higher percentile, 99.9th and so forth. Gil Tene has an excellent talk1 on this. 1 link to this video

Rafuinoon Hacker News • Recommends

On “Averages Can Be Misleading: Try a Percentile (2014)”

This topic always leads me to think about this great talk from Gil Tene on how NOT to measure latencies (basically, don't use averages!). link to this video I'm also a huge fan of how Dormando showed latency distributions in one of his recent Memcached Extstore posts. The default is 95th percentile but you can change the percentile to what matters to you (i.e. 99th percentile if you ask me!). Scroll down to see what he did and play with it. link

wmfivon Hacker News • Recommends

On “Ask HN: Must Read from ACM Library?”

This a life changing video by Gil Tene on this topic. Bottom line is because of the number requests to support a single customer "request", the high percentiles are the ones that actually matter. link to this video

leetrouton Hacker News • Recommends

On “Grasshopper – An Open Source Python Library for Load Testing”

We use locust at work but I HIGHLY recommend wrk for a very robust yet simple load testing tool. link And of course, this talk by Gil Tene is fantastic if you're interested in load testing stats link to this video

kbon Lobsters • Recommends

"How NOT to Measure Latency" by Gil Tene

programming-talkscurated list on GitHubRecommends

Under “Web Development”

How NOT to Measure Latency 00:42:59 by **Gil Tene** (2015)

telotortiumon Hacker News • Recommends

On “Unlearning descriptive statistics”

Gil Tene has an entire talk about this point called "How NOT to measure latency" ( link to this video ). In many circumstances, a web service (either server or client) has large numbers of dependencies, so a user only has to hit 95th-percentile latency for one dependency to have their overall latency significantly hurt. In other words, users may encounter the "rare" bad case on a majority of requests. Thus, even higher percentiles should be tightly controlled in order to truly keep a web service's overall latency under control.

latchon Hacker News • Recommends

On “We can do better than percentile latencies”

Worth watching is How NOT to Measure Latency: link to this video The speaker Gil Tene is also the author of the HdrHistogram which addresses this articles point: link

allanrboon Hacker News • Recommends

On “Lies, Damned Lies, and Averages: Perc50, Perc95 Explained for Programmers”

The "how not to measure latency" talk by Gil Tene is another really good explanation of this topic: link to this video

agallegoon Hacker News • Recommends

On “Kafka Removing Zookeeper Dependency”

who cares about lower percentage tiers in the world of big data? what does the p30 or 50 even matter at large scale? See Gil Tene's talk on how not to measure latency. link to this video What matters is what most of your customers will get - p99, p999, p9999, p100.

super_linearon Hacker News • Recommends

On “How Not to Measure Latency (The P99 Lie) by GIL Tene”

Tried to link to the specific timestamp in the video, where the p99 lie is discussed. Core takeaway for me was that most users of a page will experience the p99 latency of the system: link to this video

xxporon Hacker News • Disputes

On “Firefox Got Faster for Real Users in 2023”

I'm sure internally folks are looking at more stuff, but blog post wise, it makes it look like you're intentionally omitting it (see link to this video )

dangon Hacker News • Mentions

On “How Not to Measure Latency”

This is a summary of link to this video . It looks like a good summary, so I guess we won't replace the URL above, even though HN prefers original sources. We did change the title to that of the talk though.

hackercomplexon Hacker News • Mentions

On “Tech-Hub Housing Costs”

this is like.. average latency of 600ms.. sounds great when you tell your CEO but it still doesn't really tell the true story about how a huge portion of your customers are waiting 1.2s for their goddamn page to load. Max doesn't tell the whole story either, but it's pretty revealing. Show me that 99.9999 percentile baby! 4 point precision then we're talkin.. and none of this Coordinated Omission bullshit. link to this video </rant>

sitkackon Hacker News • Mentions

On “Sanic – Python 3.5+ web server that's written to go fast”

You almost never want to use average as your metric when dealing with time. For time, the 95 or 99 percentile latency is going to be measure of how many requests out of a hundred a less than your metric. Esp when the thing you are measuring is a piece of larger system, which is the crux, because of how a resource in turn requests k-more resources. Each request has an equal chance of being over the latency percentile. Even at 10 requests, only 90% of sessions will be under the 99% latency. See also, Gil Tene link to this video A nice overview, link See also, link

the8472on Hacker News • Mentions

On “Deterministic thinking: a problem in how we think, not just in how we act”

Some additional examples of people fixing instances of this problem Dissolving the fermi paradox 0, convolves distributions instead of multiplying point estimates of life/no life. How to NOT measure latency , looks at CDFs instead of slow/fast thresholds. 0 link 1 link to this video