Tuning Druid Clusters at Scale | ironSource, Lyft, Imply

Описание: ironSource, Lyft, and Imply are diving deeper into the newest real-time analytics database on the market: Druid. We’ll discuss how to tune druid clusters at high scale (several million events per second) and how to run queries quickly that can handle this high traffic. You’ll hear from our expert speakers from ironSource, Lyft, and Imply about how each company deploys druid and creates the best architectures for this cutting-edge technology.

The agenda:

The Rise of Immediate Intelligence
Rachel Pedreschi, VP of Community & Developer Relations at Imply
Decision making is changing: Apache Druid is a new type of database for creating the next generation of analytics applications that maximize flexible exploration over fresh, fast-arriving data. In this talk, Rachel Pedreschi introduces these new "immediate intelligence" applications, tells the story of Druid's emergence, and describes how data pipelines built with Druid differ from those you may already be familiar with.

Know Your Data
Jonathan Kaplan, Data Infrastructure at ironSource
The performance of our largest internal Druid cluster (in terms of incoming traffic) started to degrade, and tuning Druid infrastructure parameters didn’t work. It forced us to take a different approach, which we call "Data first, Tuning second".

Making Druid Realtime
Elad Eldor, Data Infrastructure at ironSource
In our busiest internal Druid cluster (in terms of concurrent queries), queries were very slow. We’ll describe how query performance significantly improved by tuning the Druid infrastructure.

Building Data Pipelines using Druid at Lyft
Tianyu Hong, Software Engineer, Data Infrastructure Team at Lyft
In this talk, we'll learn more about how Lyft builds data pipelines using Apache Druid, which is useful for several use cases including metrics tracking, model forecasting, and internal tools. We'll also talk about the challenges we faced while setting up our real-time ingestion pipeline into Druid using Apache Flink and Kafka, and how we went about solving them.