1000+ Columns and 5ms Latency: Building a Data Lakehouse for Flight Simulation with Apache Iceberg

Описание: The digital transformation of industry drives a crucial need for effective data management, especially for real-time applications. While many existing C++ solutions excel at monitoring and precisely tracking industrial activities, they struggle to historize data, hindering long-term analysis.

This talk presents a concrete experience report on how a leader in flight simulation overcame this challenge with Apache Iceberg. Faced with the complexity of digitizing a simulation process, this client had flight recordings, but lacked a digital representation of each simulation event.

We will detail the approach adopted to capture data from an existing C++ solution and integrate it with Apache Iceberg. This data, generated and captured in 5ms cycles, was integrated into Iceberg via a technical solution including the Apache Iceberg Sink Connector. The latter efficiently managed schemas with over 1000 columns, while ensuring real-time data capture and transmission, and data integrity.

This data capture and its restructuring for analytical purposes transforms flight simulation! The semantic layers of Dremio facilitate data access. New doors are opening: intelligent agents rely on this data, both in motion and at rest, to completely rethink the simulation exercise. They leverage both real-time and historical analysis to improve pilot training and optimize simulation execution. This digitalization example is very helpful in the brick and mortar sectors.