Zero to Scale: Telemetry pipeline with Apache Cassandra

Описание: More: https://2025.berlinbuzzwords.de/sessi...

Speaker: Shikhar Srivastava, Nomin-Erdene Oyun

Picture billions of messages pouring in daily from thousands of data providers around the globe, which are then processed and published to customers. How can one design a telemetry system to capture, publish, and then index essential information about the data flowing through the system to give internal teams visibility to aid in troubleshooting?

A core part of our business is to receive and then process humongous amounts of financial data from all over the globe. This pipeline scales to tens of billions of pricing messages every day, in which each message carries highly valuable information. Getting visibility into what was sent to us versus what was published to our customers is of utmost importance to enable internal teams to quickly troubleshoot issues reported by any of the data providers.

But how do we capture essential information about the data flowing through such a massive and high throughput system scaling to more than tens of thousands of processes running on close to a hundred machines, in which traffic peaks at more than a million messages per minute? In this talk, we will talk about how we built a high throughput telemetry system for streaming, storing, and searching such a high volume of data, starting from scratch using open source technologies like ZeroMQ, Apache Kafka, Kubernetes, and Apache Cassandra. You will gain valuable insights into the system’s design and performance, as well as the lessons we learnt along the way. We will cover everything from schema design and load testing to incremental deployment in order to manage such high data throughput.

###

Follow us on Social Media and join the Community!

Mastodon: https://floss.social/@BerlinBuzzwords
LinkedIn:   / berlin-buzzwords  
Website: https://berlinbuzzwords.de
Mail: [email protected]

Berlin Buzzwords is an event by Plain Schwarz – https://plainschwarz.com