Abstract:Standard Mixture-of-Experts (MoE) transformers route tokens to expert subnetworks within each layer, but the layer structure itself remains monolithic. We introduce Mixture of Layers (MoL), which replaces full-width transformer blocks (d_model) with K parallel thin blocks at reduced dimensionality (d_thin << d_model), connected via learned down/up projections and composed via top-k block routing. Scaling sparse block routing to many blocks creates an attention coverage problem, as each block sees fewer tokens. We address this by introducing hybrid attention, which pairs one shared softmax block for global context with Gated DeltaNet linear attention in routed blocks.
| Subjects: | Machine Learning (cs.LG); Artificial Intelligence (cs.AI) |
| Cite as: | arXiv:2605.09516 [cs.LG] |
| (or arXiv:2605.09516v1 [cs.LG] for this version) | |
| https://doi.org/10.48550/arXiv.2605.09516 arXiv-issued DOI via DataCite (pending registration) |
Submission history
From: Ivan Ternovtsii [view email]
[v1]
Sun, 10 May 2026 12:53:28 UTC (42 KB)