State Space Models (S4, S5, S6/Mamba) Explained

Описание: In this video we give a quick(ish) overview of state space models and how to use them as a layer in a neural network. We cover S4, S5 and S6/Mamba.

References I like:
S4: https://arxiv.org/abs/2111.00396, https://stacks.stanford.edu/file/drui..., https://srush.github.io/annotated-s4/
S5: https://arxiv.org/abs/2208.04933
S6/Mamba: https://arxiv.org/abs/2312.00752
Mamba as attention: https://arxiv.org/abs/2403.01590
Very nice overview of architectures and their performance on synthetic benchmarks: https://arxiv.org/pdf/2403.17844

Ps. Apologies for the dog barking in the background; need to buy a proper microphone :D