Summarizing Software Logs with Vector Search & LLMs | Open Source Experience

Описание: Free trial 👉 https://www.centreon.com/free-trial/?...

In a world where software systems generate massive volumes of logs every second, effectively summarizing this data is vital for maintaining system health. But sending millions of logs directly to an LLM is slow and prohibitively expensive.

In this talk from Open Source Experience, Sami Mourched, Data Scientist at Centreon, demonstrates how to build a scalable system that combines Approximate Nearest Neighbor Search (ANNS) with Large Language Models (LLMs) to effectively summarize gigabyte-scale daily software logs.

Sami explains the architecture of a log summarization pipeline that reduces processing costs by 90% compared to raw LLM usage.

Key topics covered:

The challenge of log analysis for SREs during incidents.

Why standard LLMs struggle with massive log volumes (Context Window & Cost).

Shingling & MinHash: Efficiently creating log signatures without fixed vocabularies.

IVF-PQ (Inverted File with Product Quantization): Fast indexing and sublinear search.

K-Means Sampling: Selecting representative logs to create a concise prompt.

LLM Agent: Generating a structured timeline of events from the sample.

About Centreon: Centreon is a leading provider of IT monitoring and observability solutions. This talk introduces features from the upcoming Centreon Log Management product.

Chapters:
00:00 - Introduction: Centreon & The Observability Platform
00:50 - The SRE Nightmare: Debugging at 3 AM
01:35 - Why sending all logs to LLMs doesn't work (Cost & Context)
02:27 - The Solution Pipeline: ANNS + LLM
03:40 - Anatomy of a Log & The Cost Challenge ($71 vs $0.03)
05:12 - Step 1: Shingling for fast vector embeddings
07:14 - Step 2: MinHash & Similarity Estimation
10:47 - Step 3: Indexing with IVF-PQ & K-Means Clustering
13:30 - Step 4: The Summarizer Agent (Prompt Engineering)
14:40 - Results: 90% Cost Reduction & Conclusion

#OpenSourceExperience #Centreon #LLM #LogManagement #AI #DataScience #SRE #DevOps #VectorSearch #Observability