Abstract:Self-attention in Transformers relies on globally normalized softmax weights, causing all tokens to compete for influence at every layer. When composed across depth, this interaction pattern induces strong synchronization dynamics that favor convergence toward a dominant mode, a behavior associated with representation collapse and attention sink phenomena. We introduce Krause Attention, a principled attention mechanism inspired by bounded-confidence consensus dynamics. Krause Attention replaces similarity-based global aggregation with distance-based, localized, and selectively sparse interactions, promoting structured local synchronization instead of global mixing. We relate this behavior to recent theory modeling Transformer dynamics as interacting particle systems, and show how bounded-confidence interactions naturally moderate attention concentration and alleviate attention sinks. Restricting interactions to local neighborhoods also reduces runtime complexity from quadratic to linear in sequence length. Empirically, we validate Krause Attention across diverse settings, including vision (ViT on CIFAR/ImageNet), autoregressive image generation (MNIST/CIFAR-10), large language models (Llama/Qwen), and language models trained from scratch at multiple scales (100M/200M). Across these domains, Krause Attention achieves consistent performance gains while improving computational efficiency, highlighting bounded-confidence dynamics as a scalable and effective inductive bias for attention.
| Comments: | ICML 2026, Project page: this https URL |
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2602.11534 [cs.LG] |
| (or arXiv:2602.11534v3 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2602.11534 arXiv-issued DOI via DataCite |
Submission history
From: Jingkun Liu [view email]
[v1]
Thu, 12 Feb 2026 03:47:53 UTC (33,943 KB)
[v2]
Wed, 18 Mar 2026 02:05:55 UTC (33,943 KB)
[v3]
Thu, 14 May 2026 03:50:30 UTC (33,947 KB)