When Kubeflow Fights Cilium: Debugging 60% Idle GPUs... Ramkumar Nagaraj & Bingi Narasimha Karthik

Ramkumar Nagaraj and Bingi Narasimha Karthik explain how conflicting scheduling decisions between Kubeflow and Cilium caused GPU underutilization and how they resolved it.

Don't miss out! Join us at our next KubeCon + CloudNativeCon events in Yokohama, Japan (29-30 July, 2026), and Shanghai, China (8-9 September, 2026) Salt Lake City, United States (Nov 9–12, 2026). Connect with our current graduated, incubating, and sandbox projects as the community gathers to further the education and advancement of cloud native computing. Learn more at kubecon.io When Kubeflow Fights Cilium: Debugging 60% Idle GPUs in Kubernetes - Ramkumar Nagaraj & Bingi Narasimha Karthik, Adobe We built a research testbed to validate ML workload scalability on Kubernetes with Kubeflow and Cilium. During 500-node stress tests, GPUs sat idle 60% of the time while pods waited for available resources. Through controlled experiments, we isolated the cause: Kubeflow's pipeline scheduler and Cilium's network-aware pod placement make conflicting decisions. Kubeflow schedules pods without considering network topology. Cilium optimizes networking but can't move scheduled pods. Result? GPUs unused while the scheduler searches for placement that won't happen. This talk shares our systematic investigation, diagnostic methodology, and scheduling constraints that resolved it. Lab tests show GPU utilization improved from 40% to 85%. You'll see the problem reproduced live, understand why it's hard to detect, and get tested Kubernetes configs. This matters for anyone planning distributed training with Kubeflow or similar orchestrators on network-optimized clusters.