Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation (Jan 2026)

Описание: Title: Self-Attention at Constant Cost per Token via Symmetry-Aware Taylor Approximation (Jan 2026)
Link: http://arxiv.org/abs/2602.00294v1
Date: January 2026

Summary:
This paper proposes a formulation of self-attention that achieves constant time and space complexity per token ($O(1)$) during inference, regardless of context length. By decomposing the Taylor expansion of the attention mechanism into symmetric chains of tensor products, the authors derive a minimal polynomial feature basis that allows for efficient feed-forward transformations. This method replaces the growing Key-Value (KV) cache with a fixed-size hidden state, enabling unbounded sequence generation with significantly reduced memory and computational requirements compared to conventional Transformers.

Key Topics:
Linear Attention
Taylor Approximation
Symmetric Tensors
Efficient Transformers
Constant Cost Inference
Polynomial Kernel Feature Maps

Chapters:
00:00 - Introduction and Authors
01:17 - The KV Cache Bottleneck
03:22 - Constant Cost Attention Overview
04:05 - Taylor Series Approximation
05:27 - Solving Dimensionality With Symmetry
07:00 - Parallel Processing Mechanics
08:00 - Fixed Size Hidden State
08:50 - The Inverse Cost Paradox
10:20 - Accuracy vs Standard Attention
11:50 - Current Implementation Limits
13:00 - Building Infinite Agents

Stock video credits:
Pressmaster - https://www.pexels.com/@pressmaster
Tom Fisk - https://www.pexels.com/@tomfisk
Yaroslav Shuraev - https://www.pexels.com/@yaroslav-shuraev
Bedrijfsfilmspecialist.nl - https://www.pexels.com/@bedrijfsfilms...
Silviu Din - https://www.pexels.com/@silviu-din-16...
Kelly - https://www.pexels.com/@kelly
Soumya - https://www.pexels.com/@soumya-1446957
Google DeepMind - https://www.pexels.com/@googledeepmind
Pixabay - https://www.pexels.com/@pixabay
Colors Motion Graphics - https://www.pexels.com/@colors-motion...
MART PRODUCTION - https://www.pexels.com/@mart-production
Ron Lach - https://www.pexels.com/@ron-lach
cottonbro studio - https://www.pexels.com/@cottonbro
Tiger Lily - https://www.pexels.com/@tiger-lily
Tima Miroshnichenko - https://www.pexels.com/@tima-miroshni...
Oleg Gamulinskii - https://www.pexels.com/@oleg-gamulins...
Nino Souza - https://www.pexels.com/@ninosouza
Danil Shostak - https://www.pexels.com/@danil-shostak...
Claudiu Ciobanu - https://www.pexels.com/@claudiuciobanu
Max Fischer - https://www.pexels.com/@max-fischer
StefWithAnF - https://www.pexels.com/@stefwithanf-1...
Engin Akyurt - https://www.pexels.com/@enginakyurt
Cyriac von Czapiewski - https://www.pexels.com/@cyriac-von-cz...
Pachon in Motion - https://www.pexels.com/@pachon-in-mot...
olia danilevich - https://www.pexels.com/@olia-danilevich
Caleb Oquendo - https://www.pexels.com/@caleboquendo
Kindel Media - https://www.pexels.com/@kindelmedia
Colin Jones - https://www.pexels.com/@larchmedia
Magda Ehlers - https://www.pexels.com/@magda-ehlers-...
Pavel Danilyuk - https://www.pexels.com/@pavel-danilyuk
crazy motions - https://www.pexels.com/@crazy-motions...
Adis Resic - https://www.pexels.com/@adis-resic-29...
Mikhail Nilov - https://www.pexels.com/@mikhail-nilov
Anthony 🙂 - https://www.pexels.com/@inspiredimages