KCD Denmark 2024: High Performance Data Backend for Your AI/ML with MinIO and Cilium Cluster Mesh

Описание: Session: High Performance Data Backend for Your AI/ML with MinIO and Cilium Cluster Mesh - George Zubrienko & Vitalii Savitskii

Owning full value chain of your product opens a vast array of possibilities to optimize operations and generate more revenue. However, utilizing commercially available tools in this area greatly diminishes the net result and doesn’t scale well with problem complexity. Thus, In ECCO we rely on highly complex demand forecasting powered by open-source LightGBM framework and employ best-in-class Mixed Integer Optimization framework Ortools from Google to fulfill online orders and replenish our stores, to have the right product in the right time, in the right place. Still, all these marvels are of little use if your data takes ages to load, or if your input datasets take hours to produce. In this talk we’ll demonstrate how embedding your storage into a cluster mesh with your Spark and ML containers via Cilium can outperform classic commercial blob storage offerings, as well showcase the reference architecture that we currently run in production. We'll also provide sample code to provision a pair of clusters (AKS/EKS) operating a shared service through Cilium Cluster Mesh