u/asdfin r/kubernetes • Recommends
Throttling: New Developments in Application Performance with CPU Limits - Dave Chiluk, Indeed
0:00 / 0:00
#cfsquota#cputhrottling#linuxkernel
Throttling: New Developments in Application Performance with CPU Limits - Dave Chiluk, Indeed
CNCF [Cloud Native Computing Foundation]
141K subscribers • 77 videos on Hallway • Conference channel
- Recommended by 6 people: 6 in engineering communities.
- A classic. People kept recommending it from 2019 to 2025.
- Presented by Dave Chiluk, Indeed
- May be out of date. Uploaded 6 years ago, in an area where tools change fast.
Intermediate. Assumes working knowledge.
27K views6 years ago6 recommendations
Systems programmingCloud and DevOpsKubernetesPerformance and optimizationOperating systems
Dave Chiluk explains how the Linux kernel CFS quota causes unexpected CPU throttling in Kubernetes and how to address it.
Join us for Kubernetes Forums Seoul, Sydney, Bengaluru and Delhi - learn more at kubecon.io Don't miss KubeCon + CloudNativeCon 2020 events in Amsterdam March 30 - April 2, Shanghai July 28-30 and Boston November 17-20! Learn more at kubecon.io. The conference features presentations from developers and end users of Kubernetes, Prometheus, Envoy, and all of the other CNCF-hosted projects Throttling: New Developments in Application Performance with CPU Limits - Dave Chiluk, Indeed Are you seeing excessively long tail response times from your applications running on containerized clouds (Kubernetes, Docker, Marathon)? Have you ever seen an application be throttled even though it’s no where near its CPU limit? Up till now, the answer has always been simply turn off hard-limits, but that has potentially nasty performance implications in shared environments. Now there's another option! This session will explain the real root cause of what has been happening. We'll introduce the kernel mechanisms that Kubernetes and other Container Orchestrators rely on to enforce CPU limits. We'll then show how they were broken, how we fixed them, and what those changes mean for you and your clouds. By the end of this session you'll understand exactly what you are getting when you set the CPU limits on your pods. sched.co/Uae1Why it's here
6 people recommend thisWhat people say. Recommended as an excellent explanation of how Linux CFS quota calculations cause unexpected CPU throttling in Kubernetes containers.
u/Rewpertousin r/kubernetes • Recommends
On “Understanding Kubernetes limits and requests by example”
This is a good start to the conversation but think others recognize this is a deeper topic as seen by talks around CPU shares: link to this video
u/rjshrjndrnin r/kubernetes • Recommends
On “k8s CPU throttling with CPU Limit”
For me, removing cpu limit should be a last resort, because if some app goes south, it shouldn't pull down entire cluster with it.
Ideally,
request = what's required to support your application's usual traffic, **Not idle state**
limit = 10-20% more for headroom
Set hpa with 60-70 pcnt cpu
This video gives great overview on what's going on[video
u/azman0101in r/kubernetes • Recommends
On “How is CPU utilisation calculated?”
The CPU Throttling issue was on Linux Kernel.
This is very well explained, by David Chiluk, in this video link to this video
He explains how CFS Quota calculation is made, by the Linux Kernel.
You don't have to remove CPU limit with a recent kernel if the throttling problem was linked to those Kernel issue.
There are metrics to monitor CPUThrottlingHigh
https://github.com/prometheus-community/helm-charts/blob/87a9d4679c81a60c2268ec9776828dc2adcd4a3d/charts/kube-prometheus-stack/templates/prometheus/rules-1.14/kubernetes-resources.yaml#L181 and then increase limits accordingly.
To my understanding, there is some edge cases in which you may need to switch for a different CPU Manager policy than the…
u/rebootsolvesthingsin r/kubernetes • Recommends
On “CPU throttling inspite of microservices consuming less than the set requests”
Could it be the limits maybe if set?
I’ve always found this video useful if it helps - oldie but a goodie link to this video
u/erewokin r/kubernetes • Recommends
On “Doubt about memory and cpu limit, is that over commitment too much or is it good to go? Containers sum shouldn't get to that number but...”
Here's where I learned about the details of this issue:
[video
u/rjshrjndrnin r/kubernetes • Recommends
On “Set optimal Kubernetes container requests in seconds using Kubecost”
I agree about the weight. Check this kubecon video. This will clarify the requests and relation with throttling.
link to this video
u/tehnicin r/kubernetes • Mentions
On “CPU Limits in Kubernetes: Why Your Pod is Idle but Still Throttled: A Deep Dive into What Really Happens from K8s to Linux Kernel and Cgroups v2”
> Dave Chiluk (src: link to this video
I assume this is the correct youtube video? link to this video
u/dbowlingin r/kubernetes • Mentions
On “Weekly: This Week I Learned (TWIL?) thread”
I finally understand enough about throttling to not second guess every decision on CPU limits thanks to this talk: link to this video
The bug he talks about is fixed, but so much makes sense that was vague before seeing this. It’s great to hear him wrap up at the end regarding what their target for throttle metrics are.