The Geometry of Projection Heads: Conditioning, Invariance, and Collapse (ICML 2026)

Описание: This work proposes a geometric theory of projection heads in self-supervised learning which builds on the existing information-theoretic literature. In this view, heads are modelled as learned Riemannian metrics on representation space which shield the backbone from the destructive constraints of the loss function.

Paper: https://arxiv.org/abs/2605.17180
Code: https://github.com/farischaudhry/proj...

Related work on spectral geometry and hyperparameter selection via simulated annealing / large deviations: https://arxiv.org/abs/2603.12552