IEX2H5: Fast IEX TOPS to HDF5 Conversion - Call the Doctor 8/5/25

Описание: During this HDF Clinic episode, The HDF Group’s Gerd Heber and Steven Varga of Varga Consulting met to discuss the challenges of processing and storing tick data, a critical component for financial analytics and algorithmic trading. Steven has just released IEX2H5, a tool designed to convert raw network traffic data in PCAP format into analytics-ready HDF5, optimizing access time and storage through compression. Using standard Gzip, compression ratios of 3:1 to 4:1 can be easily achieved. However, much higher compression ratios are possible with more bespoke compression methods.

HDF5 is perhaps the most popular format for storing large amounts of financial data due to its ability to load data into various systems, its compatibility with virtually all programming languages, its direct control over tick streams, and its longevity and archival qualities. Steven’s tool, IEX2H5, is used for backtesting and building frameworks with IEX Exchange data to extract and analyze trade counts and identify top-performing stocks from this vast dataset with high-speed processing. It represents a significant step towards data liberation and reproducibility in financial market research. Give your research a boost by using IEX2H5 and spread the word by starring the GitHub repository at https://github.com/vargaconsulting/ie...!

For additional information on processing financial data (or any other large datasets!) with HDF5, reach out to The HDF Group. https://www.hdfgroup.org/contact-us/

This session was recorded on August 5, 2025.

Join us each Tuesday at 12:20 p.m. Central time (US/Canada) by joining the zoom at https://us06web.zoom.us/j/98286880081

Call the Doctor is a series of weekly, unscripted, live events! The HDF Group’s staff members will answer attendee questions and, for example, go over the previous week’s HDF Forum posts. The HDF Clinics are free sessions intended to help users tackle real-world HDF problems from a common cold to severe headaches and offer relief where that’s possible. As time permits, we will include how-tos, offer advice on tool usage, review your code samples, teach you survival in the documentation jungle, and discuss what’s new or just around the corner in the land of HDF.