llm-d: Distributed LLM Inference on Kubernetes

Name: llm-d: Distributed LLM Inference on Kubernetes
Uploaded: 2026-06-27T00:58:03Z
Duration: 20 min 57 s
Channel: The Cef Experience
Description: An architectural overview of llm-d, a system for distributed LLM inference on Kubernetes using Gateway API and KV cache-aware routing.

The Cef Experience

1.8K subscribers • 1 video on Hallway

YouTube

Recommended by 1 person: 1 in engineering communities.

Advanced. Deep dive for people who know the field.

4.7K views4 months ago1 recommendation

AI and machine learning Distributed systems Transformers and LLMs Kubernetes Networking and protocols

An architectural overview of llm-d, a system for distributed LLM inference on Kubernetes using Gateway API and KV cache-aware routing.

Blog post: cefboud.com/posts/llm-d/ llm-d: llm-d.ai/docs/getting-started Introduction to LLMD Why LLM inference needs smarter load balancing Prefill vs Decode explained KV cache awareness and session routing How LLMD scores model servers LLMD Router architecture Client request flow Envoy External Processing (ExtProc) End-to-end request routing Gateway API Inference Extension Prefill/Decode disaggregation KV cache transfer with NCCL & RDMA Plugin architecture and extensibility Flow control, priorities & autoscaling Final thoughts

Why it's here

1 person recommends this

What people say. Recommended as an introduction to distributed LLM inference and the Gateway API Inference Extension.

u/RelevantEmergency707in r/kubernetes • 4 months agoRecommends

llm-d: Distributed LLM Inference on Kubernetes Gateway API Inference Extension

38View on Reddit