On Surprising Effectiveness of Masking Updates in Adaptive Optimizers (Feb 2026)

Описание: Title: On Surprising Effectiveness of Masking Updates in Adaptive Optimizers (Feb 2026)
Link: http://arxiv.org/abs/2602.15322v1
Date: February 2026

Summary:
This paper demonstrates that randomly masking parameter updates in adaptive optimizers can significantly improve LLM pre-training by inducing an implicit curvature-dependent geometric regularization. The authors introduce Magma (Momentum-aligned gradient masking), a drop-in wrapper that prioritizes updates based on momentum-gradient alignment. Magma yields substantial perplexity reductions and enhanced training stability across model scales, outperforming state-of-the-art optimizers like Adam and Muon while remaining robust to learning rate variations.

Key Topics:
LLM Pre-training
Adaptive Optimizers
Gradient Masking
Geometric Regularization
Optimization Stability
Magma Optimizer
Curvature Sharpness

Chapters:
00:00 - Introducing Magma Optimizer
01:24 - Testing Random Skip Updates
02:53 - Comparing Weight vs Momentum
04:23 - Filtering Gradient Noise
06:09 - Implementing Block-Wise Masking
07:45 - Explaining Geometric Regularization
09:18 - Managing Heavy-Tailed Noise
10:42 - Analyzing LLM Benchmarks
12:08 - Scaling Mixture of Experts
13:35 - Enhancing Learning Rate Robustness
15:13 - Maintaining Dense History Buffers
16:51 - Evaluating Future Training Efficiency

Stock video credits:
Silviu Din - https://www.pexels.com/@silviu-din-16...
José Alfredo Munguía Lira - https://www.pexels.com/@rectorretro
cottonbro studio - https://www.pexels.com/@cottonbro
Google DeepMind - https://www.pexels.com/@googledeepmind
Pressmaster - https://www.pexels.com/@pressmaster
Soumya - https://www.pexels.com/@soumya-1446957
Bedrijfsfilmspecialist.nl - https://www.pexels.com/@bedrijfsfilms...
Tom Fisk - https://www.pexels.com/@tomfisk
Mikhail Nilov - https://www.pexels.com/@mikhail-nilov
Charlie Mounsey - https://www.pexels.com/@charlie-mouns...
Stas Knop - https://www.pexels.com/@stasknop
Danil Shostak - https://www.pexels.com/@danil-shostak...
Dan Cristian Pădureț - https://www.pexels.com/@paduret
Oleg Gamulinskii - https://www.pexels.com/@oleg-gamulins...
StefWithAnF - https://www.pexels.com/@stefwithanf-1...
Ron Lach - https://www.pexels.com/@ron-lach
Trippy Lagoon - https://www.pexels.com/@trippy-lagoon...
Pachon in Motion - https://www.pexels.com/@pachon-in-mot...
Nino Souza - https://www.pexels.com/@ninosouza
Anete Lusina - https://www.pexels.com/@anete-lusina
Colors Motion Graphics - https://www.pexels.com/@colors-motion...
tunnel motions - https://www.pexels.com/@tunnelmotions
Colin Jones - https://www.pexels.com/@larchmedia
Pavel Danilyuk - https://www.pexels.com/@pavel-danilyuk
Yaroslav Shuraev - https://www.pexels.com/@yaroslav-shuraev
Kindel Media - https://www.pexels.com/@kindelmedia
olia danilevich - https://www.pexels.com/@olia-danilevich
fauxels - https://www.pexels.com/@fauxels
Tiger Lily - https://www.pexels.com/@tiger-lily
Ketut Subiyanto - https://www.pexels.com/@ketut-subiyanto
Adis Resic - https://www.pexels.com/@adis-resic-29...
@svetjekolem - https://www.pexels.com/@svetjekolem
crazy motions - https://www.pexels.com/@crazy-motions...
Pixabay - https://www.pexels.com/@pixabay
Caleb Oquendo - https://www.pexels.com/@caleboquendo