Observability in AI apps. Eval Engineering for AI Developers, lesson 2 - add observability to AI

Описание: Learn Eval Engineering in this free, 5-part, hands-on course presented by ‪@jimbobbennett‬

​90% of AI agents don't make it successfully to production. The biggest reason is the AI engineers building these apps don't have a clear way of evaluating that these agents are doing what they should do, and using the results of this evaluation to fix them.

​In this course, you will learn all about evals for AI applications. You'll start with some out-of-the-box metrics and learn about evals, then move onto understanding observability for AI apps, analyzing failure states, defining custom metrics, then finally using these across your whole SDLC.

​This will be hands on, so be prepared to write some code, create some metrics, and do some homework!

​In this second lesson, you will

Use observability to visualize the components of a typical multi-agent AI application
Learn about the different components that make up these applications
Apply some out-of-the-box metrics to start to get an understanding of how your application is working
​
Prerequisites:

​A basic knowledge of Python
​- Access to an OpenAI API key
​A free Galileo account (we will be using Galileo as the evals platform). Sign up at https://galileo.ai/sign-up.
Course materials from https://github.com/rungalileo/eval-en...

Catch the rest of the lessons here:    • Eval Engineering for AI Developers  

0:00:10 - Introduction & Welcome
0:05:19 - Course Schedule & Overview
0:07:22 - Prerequisites & Setup
0:10:39 - Homework Review: Context Adherence
0:13:54 - Introduction to Observability
0:16:35 - Demo: Runzi Multi-Agent App
0:25:46 - What is Observability?
0:26:46 - Components of Observability: Spans, Traces, Sessions & Metrics
0:35:05 - Demo: Finding Failures with Observability
0:47:09 - Evaluations & Observability
0:54:33 - Adding Metrics: Instruction Adherence & Tone
1:19:36 - When to Add Metrics
1:23:08 - Homework: Breaking Runzi & Custom Metrics
1:26:41 - Q&A Session