Production-Ready Distributed Inference with Ray Serve | Anyscale | Ray Summit 2026

Описание: "Github repository: https://github.com/anyscale/ray-summi...

Design scalable distributed inference systems for production AI applications, from classic ML models to large language models.

This hands-on training session covers building Ray Serve applications with deployments, composition, and load-aware request routing, hardening for production with autoscaling, fault tolerance, and zero-downtime upgrades, deploying and scaling LLM endpoints with Ray Serve LLM on engines like vLLM and SGLang, and optimizing the inference engine with the KV cache, paged attention, prefix caching, and continuous batching.

You'll leave knowing how to scale frontier models with KV-cache-aware routing, prefill/decode disaggregation, and wide expert parallelism for mixture-of-experts models.

Liked this video? Check out other Ray Summit training session recordings:    • Ray Summit 2026 - Training Sessions  

Subscribe to our YouTube channel to stay up-to-date on the future of AI:    / anyscale  

🔗 Connect with us:
LinkedIn:   / joinanyscale  
X: https://x.com/anyscalecompute
Website: https://www.anyscale.com/"