Scaling Apache Spark at OpenAI

Описание: ​In this talk, Chao Sun (OpenAI) shares their experiences operating Apache Spark at scale within OpenAI’s data platform. He discusses how they run both Databricks Spark and self-hosted open-source Spark in parallel, with a deeper focus on the latter - covering key areas such as cluster management, job submission architecture, access control, and dynamic scaling.

He also highlights how Spark and Delta Lake power some of their most critical data pipelines, the challenges they’ve faced in building and maintaining them, and the approaches that have made Spark a reliable and efficient foundation for large-scale data processing at OpenAI.

This talk was part of the “Open Lakehouse + AI Mini Summit,” hosted at the Databricks Mountain View office on November 13, 2025.

00:00 — Intro and agenda
01:20 — OpenAI data platform
03:40 — Why two Spark stacks
06:10 — Multi‑cluster architecture and Inglis
09:10 — UC integration, caching, and ACLs
12:00 — Maglev ingestion framework
15:10 — Optimizations and big wins
19:30 — Pain points and roadmap