A Walkthrough of Progress Measures for Grokking via Mechanistic Interpretability: What? (Part 1/3)

Описание: Part 1 of a walkthrough of our paper, Progress Measures for Grokking via Mechanistic Interpretability. I'm joined by my co-author Lawrence Chan. In this part, we give an overview of the paper and discuss the key takeaways

Part 2:    • A Walkthrough of Progress Measures for Gro...  
Part 3:    • A Walkthrough of Progress Measures for Gro...  

If you want to learn more about mechanistic interpretability, check out https://neelnanda.io/getting-started

Our paper: https://arxiv.org/abs/2301.05217
Original grokking paper: https://arxiv.org/abs/2201.02177
AdamW: https://pytorch.org/docs/stable/gener...
Walkthrough of toy models of superposition:    • A Walkthrough of Toy Models of Superpositi...  
Danny Hernandez paper on scaling laws for repeated data: https://arxiv.org/abs/2205.10487
Jermyn & Schlegeris on S-Shaped Curves: https://www.alignmentforum.org/posts/...
Unifying Grokking and Double Descent: https://arxiv.org/abs/2303.06173
Omnigrok: https://arxiv.org/abs/2210.01117

0:00 - Intro
0:50 - What is grokking?
9:53 - Mechanistic interpretability
11:47 - Paper overview, modular addition algorithm
15:08 - Progress measures
21:41 - why this work is bullshit
29:30 - Predicting when it will grok?
33:45 - Why does grokking happen?
40:27 - Lottery ticket hypothesis
42:43 - Conclusion