November Data: Unlocking Observability with ClickHouse Columnar Storage

Описание: November's data revolution! Discover why ClickHouse is the ultimate solution for observability, offering unparalleled performance for logs, metrics, and traces. See ClickHouse’s advantages, use cases and unlock the power of columnar storage for your data needs.

This video explores how ClickHouse, an open-source columnar database, provides a superior alternative to traditional search engines and metric stores for handling large-scale observability data. We dive deep into its architecture, benefits, and practical applications within the Clickstack observability stack. This video will premiere on 2026-02-01 11:00 Europe/Amsterdam.

What You'll Learn:
• Understand the limitations of search engines and metric stores for observability.
• Discover the power of columnar storage and ClickHouse for handling logs, metrics, and traces.
• Learn how Clickstack leverages ClickHouse, HyperDX, and OpenTelemetry for a complete observability solution.

TIMESTAMPS/CHAPTERS:
0:00 Introduction
2:15 ClickHouse Overview
5:30 Clickstack Explained
8:45 HyperDX and ClickHouse
12:00 Data Store Options
15:45 Search Engine Limitations
19:00 Metric Store Drawbacks
22:30 Key Requirements for a Data Store
26:00 Columnar Storage Explained
30:00 Column Storage Benefits
34:00 Stack Overflow Example
38:00 JSON Support in Column Stores
42:00 S3 Storage Integration
46:00 ClickHouse Advantages
50:00 SQL and Observability
54:00 OpenTelemetry Integration
58:00 UI Design for Optimal Performance
1:02:00 Query Optimization

Detailed Topics Covered:
We begin by examining the challenges of using traditional search engines like OpenSearch and Elasticsearch for observability, highlighting their limitations in aggregation and high cardinality data. Then, we explore metric stores like Prometheus, noting their constraints in search capabilities and exploratory workflows. Next, we delve into the core principles of columnar storage, explaining how ClickHouse’s architecture enables efficient compression, fast aggregations, and parallel query execution. The video further discusses ClickHouse's support for JSON data, integration with S3 object storage, and schema-on-read/write capabilities. We also explore real-world examples and use cases, demonstrating how ClickHouse can be used to build powerful observability stacks for managing logs, metrics, and traces.

Key Insights & Takeaways:
ClickHouse offers a cost-effective and scalable solution for storing and analyzing observability data, surpassing the limitations of search engines and metric stores. Columnar storage enables fast aggregations, efficient compression, and parallel query execution. The integration of ClickHouse with tools like HyperDX and OpenTelemetry creates a comprehensive and open-source observability stack. By understanding the principles of columnar storage and its benefits, you can optimize your data infrastructure for improved performance and scalability.

Additional Resources:
ClickHouse GitHub:
HyperDX:
OpenTelemetry:

About This Channel:
Luca Berton is dedicated to providing practical, hands-on tutorials and insightful content on data engineering, cloud computing, and data science. Subscribe for weekly videos exploring cutting-edge technologies and real-world applications, with the goal of helping you master modern data tools.

If you found this video helpful, please give it a like, leave a comment with your questions or thoughts, and subscribe to the channel for more content on data engineering and observability! Don't forget to hit the notification bell so you never miss an upload.