State-Space Models and Efficient Long-Sequence Architectures Beyond the Transformer

Abstract

The self-attention mechanism that powers modern Transformers scales quadratically in time and memory with sequence length, making very long context expensive and often impractical. This survey reviews the emerging family of state-space models (SSMs) and related sub-quadratic architectures that aim to preserve sequence-modeling quality while achieving linear or near-linear cost. Beginning with the complexity wall faced by recurrent networks and attention, the article traces the deep SSM line from the HiPPO theory of continuous memory through the Structured State Space sequence model (S4), its diagonal simplifications, and the parallel-scan formulation of S5. It then examines selective SSMs and Mamba, whose input-dependent parameters and hardware-aware parallel scan deliver linear-time training and constant-memory autoregressive inference. Parallel developments in linear attention, RWKV, RetNet, and the exact but memory-efficient FlashAttention kernel are placed in context, along with hybrid designs such as Jamba that interleave attention and SSM blocks, and the state-space duality that unifies both views in Mamba-2. Applications spanning language, audio, genomics, and vision are summarized. The survey closes with an honest account of empirical trade-offs, including the associative-recall gap between recurrent SSMs and attention, and outlines open challenges. Reported numbers are illustrative and labeled as representative rather than definitive.