Attention, Dynamic Wave and Hybrid Sequence Mixers in Language Models
Large Language Models (LLMs) rely on Multi-Head Attention (MHA) for sequence modeling, but scaling to long contexts is constrained by the quadratic cost of attention, motivating sub-quadratic recurrent alternatives.
Explainer · on the wire
The explainer for this story is still on the wire.
The claim, the mechanism, and the figures land here once generated.
Source · arXiv