Decoupling Metadata: Leveraging Queryable Iceberg Tables for Scalable, Cross-Engine Innovation

Описание: #icebergSummit 2025 breakout session delivered by Karthic Rao, #e6data.

Session Description:
As data volumes and the number of underlying parquet files continue to explode—especially in fast streaming ingestion scenarios—managing and efficiently querying metadata becomes a critical challenge. In this talk, we explore the advantages of storing vast volumes of parquet metadata statistics from both Delta and Iceberg tables as flat, denormalized, and partitioned queryable Iceberg tables. This innovative approach ensures lightning-fast query performance when metadata is appropriately partitioned and fundamentally decouples the metadata layer from traditional query engines.

By converting metadata into a queryable dataset, we open up a wealth of opportunities for independent innovation across various components of the lakehouse ecosystem. Query planners, cost estimators, query guardrails, and compaction services can all access rich, accurate statistics directly—allowing each to iterate independently without waiting on monolithic engine releases. This separation of concerns simplifies scalability and levels the playing field for diverse query engines, enabling each to optimize and plan queries based on a unified, high-performance metadata store.

During the session, I will share our real-world experience managing approximately 200GB of Delta table parquet stats stored as an Iceberg table. This case study demonstrates how this approach improved query engine performance, providing enhanced flexibility across teams and streamlined metadata-driven innovations. We will also discuss how Delta and Iceberg table metadata can be ingested into queryable Iceberg tables—paving the way for a standardized interface.

Recent discussions at the Apache PMC level around serving metadata through a standard interface via the Iceberg Catalog API highlight the community’s growing interest in decoupled metadata management https://lists.apache.org/thread/jbg14.... I will present our vision for this standardization, outline the benefits of having metadata available in a queryable form, and call upon the community to share their insights. This collaborative effort could revolutionize the design of modern data platforms, driving significant improvements in query planning, execution, and overall system scalability.

Join us as we delve into:

•Scalability & Performance: How flat, denormalized, and well-partitioned Iceberg tables enable ultra-fast metadata queries even when dealing with tens of millions of parquet files.
•Cross-Component Innovation: The decoupling of metadata access from query engines allows independent innovation in planners, guardrails, cost estimators, and compaction strategies.
•Real-World Learnings: Insights from managing 200GB of Delta table metadata and the resulting performance gains and operational flexibility.
•A Call to Standardize: Exploring the potential of a unified metadata serving interface via the Iceberg Catalog API and inviting community feedback to shape the future of lakehouse architectures.

This talk is a must-attend for anyone interested in the next frontier of metadata management and the decoupling of computing services within the modern data ecosystem.