TMLW02 | Martin Keller-ressel | Emergence of heavy tails in homogenised stochastic gradient descent

Описание: TMLW02 | Prof. Martin Keller-ressel | Emergence of heavy tails in homogenised stochastic gradient descent

Speaker: Professor Martin Keller-ressel (Technische Universität Dresden)
Date: 25th Apr 2024 - 15:15 to 16:00
Venue: External
Title: Emergence of heavy tails in homogenised stochastic gradient descent
Event: (TMLW02) SGD: stability, momentum acceleration and heavy tails
Abstract: We analyze a continuous diffusion approximation of SGD, called homogenized stochastic gradient descent, show that it behaves asymptotically heavy-tailed, and give explicit upper and lower bounds on its tail-index. We validate these bounds in numerical experiments and show that they are typically close approximations to the empirical tail-index of SGD iterates. In addition, their explicit form enables us to quantify the interplay between optimization hyperparameters and the tail-index. Our results show that also continuous diffusions, not only Lévy-driven SDEs, can accurately represent the emergence of heavy tails in SGD. In addition, our results suggest skew Student-t-distributions, not alpha-stable distributions, as surrogates of parameter distributions under SGD.

👉 Want to learn more about the talks in this stream?

Seminar Link:https://www.newton.ac.uk/seminar/42466/

Workshop Link: https://www.newton.ac.uk/event/TMLW02 /
-------------------

FOLLOW US
🌐| Website: https://www.newton.ac.uk
🎥| Main Channel:    / @isaacnewtoninstitute  
🐦| Twitter:   / newtoninstitute  
💬| Facebook:   / newton.institute  
📷| Instagram:   / isaacnewtoninstitute  
🔗| LinkedIn:   / isaac-newton-institute-for-mathematical-sc...  

SEMINAR ROOMS
🥇| INI Seminar Room 1:    / @iniseminarroom1  
🥈| INI Seminar Room 2:    / @iniseminarroom2  
🛰️| INI Satellite Events:    / @inisatellite  

ABOUT
The Isaac Newton Institute is a national and international visitor research institute. It runs research programmes on selected themes in mathematics and the mathematical sciences with applications over a wide range of science and technology. It attracts leading mathematical scientists from the UK and overseas to interact in research over an extended period.

👉 Learn more about us and our events here: https://www.newton.ac.uk