Serving the agentic workload | Simon Mo CEO at Inferact, Runtime 2026

Описание: Simon Mo, co-founder and CEO of Inferact and core maintainer of vLLM, explains what it takes to serve production agentic AI workloads as multi-turn, tool-calling systems reshape the demands placed on inference infrastructure.

He walks through the shift from single-turn chat to complex agent interactions, breaking down the challenges of prefix caching pressure, long-context latency budgets, and global session routing. He details how vLLM addresses these hurdles with hybrid KV cache allocation, distributed cache pools via Mooncake, and specialized megakernels across NVIDIA, AMD, and Google TPU hardware. He also previews the future vLLM roadmap for programmable KV caches, sub-agent hints, and agentic-first serving.

Timestamps

00:00 Introduction & Welcome
00:32 What is vLLM? Ecosystem & Community Growth
01:48 Evolution of Workloads: From Single-Turn to Agentic Interactions
03:12 How Agentic Workloads Reshape the Serving Problem
04:37 Where It Gets Hard: Prefix Caching, Execution Efficiency & Routing
05:27 Hybrid KV Cache Management (LCM Page Sizing for Sparse/Linear Attention)
07:20 Distributed & Disaggregated KV Cache Pool (Mooncake)
08:58 vLLM Agentic Performance & Pareto Frontier Benchmarks
10:58 Next Steps for vLLM: Agentic-First, Efficient & Friendly
13:00 Hardware Heterogeneity: NVIDIA, AMD MI355 & Google TPU
14:02 First TPU Megakernel: Performance & Architecture Deep Dive
15:56 vLLM Q4 Roadmap & Specialized Production Serving
18:14 Conclusion & Joining the Community