#406

Описание: Every alignment method so far runs on human judgement, and people are slow, expensive and inconsistent. In Video 406 of AI from Zero to Today, we let a model do the labelling: an AI judge guided by written principles, how often it agrees with people, how it fixes a reward model's shortcut, and the biases it brings. Can AI help align AI?

In this video:
• RLAIF: a model, not a person, decides which of two answers is better (Lee et al. 2023)
• A tiny judge with three written principles, and how often it agrees with simulated humans: 83.3%
• The judge agrees with the labels about as often as two humans agree with each other, a pattern Zheng et al. 2023 found for real model judges
• Constitutional AI (Bai et al. 2022): self-critique and revision, then AI comparisons
• Cheap AI-labelled counterexamples that remove a reward model's politeness shortcut
• Position bias: swapping the order flips 23.3% of the judge's verdicts, and the defences

⏱ Chapters
0:00 Introduction
0:25 Opening question
0:55 Recap
1:31 1. AI Feedback
3:49 2. Constitutional AI
5:02 3. Benefits and Biases
6:24 Twist question
7:48 Summary board
8:29 Goal check
9:12 Try it yourself
9:32 Next video
9:51 Outro

📚 About this series
AI from Zero to Today builds the whole subject of AI from nothing: the ideas, the mathematics and the code, until you can build real AI yourself. One idea per video, explained completely, in simple English.

▶ DriveLink: https://drive.google.com/drive/folder...

🔔 Subscribe to LetsLearn AI for new AI lessons every week

💬 Questions? Leave them in the comments. I read them all.

#RLAIF #ConstitutionalAI #LearnAI