Draft:State space model

Review waiting, please be patient. This may take 2 months or more, since drafts are reviewed in no specific order. There are 1,999 pending submissions waiting for review. If the submission is accepted, then this page will be moved into the article space. If the submission is declined, then the reason will be posted here. In the meantime, you can continue to improve this submission by editing normally. Where to get help If you need help editing or submitting your draft, please ask us a question at the AfC Help Desk or get live help from experienced editors. These venues are only for help with editing and the submission process, not to get reviews. If you need feedback on your draft, or if the review is taking a lot of time, you can try asking for help on the talk page of a relevant WikiProject. Some WikiProjects are more active than others so a speedy reply is not guaranteed. How to improve a draft Wikipedia:Contributing to Wikipedia – a basic overview on how to edit Wikipedia. Help:Wikitext – how to use the markup Help:Referencing for beginners – how to include references Wikipedia:Article development – how to develop your article Wikipedia:Writing better articles – how to improve your article Wikipedia:Verifiability – make sure your article includes reliable third-party sources You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Add tags to your draft Editor resources Easy tools: Citation bot (help) | Advanced: Fix bare URLs Reviewer tools Instructions · What links here · State space model (talk: + · bio) · (log) · Copyvios report · reFill · Citation Bot · (Search: Google, Wikipedia) · Submitted 18 hours ago by Toddphen (talk: D · +) · Last edited 18 hours ago by Toddphen Warning: The page State space modelState space model redirects to State-space representation. Please ensure it is not a copy or that this page is located at the correct title.

Submission declined on 9 March 2026 by Nighfidelity (talk). This draft appears to contain text generated by a large language model (such as ChatGPT). You cannot use LLMs to generate article content. LLM-generated pages with certain obvious signs of being machine generated may be deleted without notice. Instead, only summarize in your own words a range of independent, reliable, published sources that discuss the subject. See the advice page on large language models for more information. If you would like to continue working on the submission, click on the "Edit" tab at the top of the window. If you have not resolved the issues listed above, your draft will be declined again and potentially deleted. If you need extra help, please ask us a question at the AfC Help Desk or get live help from experienced editors. Please do not remove reviewer comments or this notice until the submission is accepted. Where to get help If you need help editing or submitting your draft, please ask us a question at the AfC Help Desk or get live help from experienced editors. These venues are only for help with editing and the submission process, not to get reviews. If you need feedback on your draft, or if the review is taking a lot of time, you can try asking for help on the talk page of a relevant WikiProject. Some WikiProjects are more active than others so a speedy reply is not guaranteed. How to improve a draft Wikipedia:Contributing to Wikipedia – a basic overview on how to edit Wikipedia. Help:Wikitext – how to use the markup Help:Referencing for beginners – how to include references Wikipedia:Article development – how to develop your article Wikipedia:Writing better articles – how to improve your article Wikipedia:Verifiability – make sure your article includes reliable third-party sources You can also browse Wikipedia:Featured articles and Wikipedia:Good articles to find examples of Wikipedia's best writing on topics similar to your proposed article. Improving your odds of a speedy review To improve your odds of a faster review, tag your draft with relevant WikiProject tags using the button below. This will let reviewers know a new draft has been submitted in their area of interest. For instance, if you wrote about a female astronomer, you would want to add the Biography, Astronomy, and Women scientists tags. Add tags to your draft Editor resources Easy tools: Citation bot (help) | Advanced: Fix bare URLs Declined by Nighfidelity 5 months ago. Last edited by Toddphen 18 hours ago. Reviewer: Inform author. This draft has been resubmitted and is currently awaiting re-review.

Submission declined on 7 December 2025 by I2Overcome (talk). This draft appears to contain text generated by a large language model (such as ChatGPT). You cannot use LLMs to generate article content. LLM-generated pages with certain obvious signs of being machine generated may be deleted without notice. Instead, only summarize in your own words a range of independent, reliable, published sources that discuss the subject. See the advice page on large language models for more information. Declined by I2Overcome 8 months ago.

State Space Models (SSMs) are a class of neural network architectures for processing time series data that model sequences using principles from control theory. SSMs have emerged as efficient alternatives to Transformer and recurrent neural network (RNN) architectures, particularly for handling long-range dependencies in sequence modeling tasks.^[1] Unlike Transformers which have quadratic complexity with respect to sequence length, SSMs achieve linear or near-linear time complexity, making them particularly effective for processing very long sequences.^[2][3]

State Space Models in deep learning are based on continuous-time state space representations from classical control theory.^[2] At their core, SSMs map a one-dimensional input signal u(t) to an output signal y(t) through a hidden state x(t) using a system of differential equations.^[4] The basic SSM is defined by the equations:

x'(t) = Ax(t) + Bu(t)
y(t) = Cx(t) + Du(t)

where A is the state matrix, B is the control matrix, C is the output matrix, and D is a direct feedthrough term (often treated as a skip connection in deep learning applications).^[1]

SSMs offer several key advantages: they can naturally handle continuous data, automatically adapt to different sampling rates without retraining, and provide mathematically tractable analysis of their dynamics.^[2][5] Through discretization, SSMs can be viewed from three complementary perspectives: as continuous-time systems, as recurrent networks during inference, and as convolutional models during training.^[1]

The application of state space models to deep learning traces back to theoretical neuroscience research. In 2018, Aaron R. Voelker and Chris Eliasmith from the University of Waterloo proposed that the dynamic system in SSMs can effectively model "time cells" present in the hippocampus and cortex, leading to their work on applying SSMs to neural networks.^[6][2]

The Legendre Memory Unit (LMU), introduced by Voelker, Kajić, and Eliasmith in 2019, was among the first successful applications of SSMs in deep learning.^[7] LMUs are mathematically derived to orthogonalize continuous-time history by solving coupled ordinary differential equations, with their phase space mapping onto sliding windows of time via Legendre polynomials. LMUs demonstrated the ability to handle temporal dependencies spanning 100,000 time steps and achieved state-of-the-art performance on permuted sequential MNIST, exceeding 97% accuracy.^[7]

The High-Order Polynomial Projection Operators (HiPPO) framework, introduced by Gu et al. in 2020 who were from Stanford University, provided a unified mathematical foundation for memory in sequence models.^[8] HiPPO optimally projects continuous signals onto polynomial bases, yielding linear dynamics for the projection coefficients. This framework produces several instantiations including HiPPO-LegS (scaled Legendre) and HiPPO-LegT (translated Legendre), which achieve timescale robustness and bounded gradients.^[8][9] The HiPPO framework achieved 98.3% accuracy on permuted MNIST, surpassing previous RNN approaches by a significant margin.^[8]

Chilkuri and Eliasmith proposed and demonstrated a method to efficiently train SSMs in parallel on GPUs.^[10] This overcomes concerns that the recurrence in SSMs would be difficult to train on GPUs, since other recurrent networks like LSTMs fell out of favour for this reason. Subsequently the first large language model (LLM) using SSMs was demonstrated to scale better than either LSTMs or Transformers using this method.^[11]

The Structured State Space sequence model (S4), introduced by Gu, Goel, and Ré in 2021, marked a breakthrough in making SSMs practical for large-scale deep learning.^[12] S4 addressed the computational challenges of naive SSM implementations through a novel parameterization involving Structured initialization which uses the HiPPO matrix for the state matrix A. As well, the model introduced a Normal plus low-rank (NPLR) decomposition, which allows A to be diagonalized stably. Finally, the model reduces the SSM to a Cauchy kernel computation to improve computational efficiency.^[12]

S4 achieved interesting results across multiple domains:^[12][13]

1. 91% accuracy on sequential CIFAR-10 (matching 2D ResNets with no data augmentation)
2. State-of-the-art on all tasks in the Long Range Arena benchmark
3. First model to solve the Path-X task (16,000 sequence length) with 88% accuracy
4. 60× faster generation than Transformers on language modeling

The model demonstrated the ability to handle sequences exceeding 10,000 steps while maintaining linear scaling in sequence length.^[12]

Mamba, introduced by Gu and Dao in December 2023, represents a major advancement in SSM architectures through the introduction of selective state space models. Earlier language-focussed SSMs use time-invariant parameters^[11], meaning the matrices A, B, and C remain constant across the sequence. Mamba's main innovation is making these parameters functions of the input, allowing the model to selectively propagate or forget information based on content^[14]. In addition, compared to previous LLM work, Mamba provided a simplified architecture that replaces attention and MLP blocks with a unified SSM block^[15] Finally, Mamba includes specific hardware-aware algorithms including parallel scan, kernel fusion, and selective recomputation to achieve efficient training^[14].

Mamba achieved competitive or superior performance compared to Transformers while providing 5× higher throughput and linear scaling to million-length sequences.^[14] On language modeling, Mamba-3B matched Transformers twice its size in both pretraining and downstream evaluation.^[14]

In May 2024, Dao and Gu introduced Mamba-2 through their "Transformers are SSMs" paper, which established theoretical connections between SSMs and attention mechanisms via structured semiseparable matrices.^[16] The State Space Duality (SSD) framework enabled the design of Mamba-2, which is 2-8× faster than Mamba while maintaining competitive performance with Transformers on language modeling.^[16]

Mamba-2 achieves faster computation by leveraging matrix multiplication primitives and tensor cores on modern GPUs, allowing for larger state expansion (typically N=128-256 compared to N=16 in Mamba) while remaining computationally efficient.^[16] The model also enables better system-level optimizations including tensor parallelism and sequence parallelism.^[16]

AI21 Labs introduced Jamba in March 2024, the first production-grade model combining Mamba SSM layers with Transformer attention and mixture-of-experts (MoE) components.^[17] Jamba features:

• A hybrid architecture interleaving Transformer and Mamba layers at a 1:7 ratio^[17]
• 52B total parameters with only 12B active parameters during inference^[17]
• Support for 256K token context windows
• 3× throughput improvement on long contexts compared to Mixtral 8x7B^[17]

The architecture demonstrated that hybrid approaches can effectively balance the strengths of both SSMs (efficiency, long context) and Transformers (performance, in-context learning).^[18] Jamba 1.5, released in August 2024, scaled to 398B total parameters with 94B active, representing the largest hybrid SSM-Transformer architecture to date.^[19]

Recent work has explored various hybrid architectures including: Vision Mamba (Vim), which has bidirectional Mamba blocks for visual data processing^[15]; MambaByte, which provides byte-level language modeling without tokenization;^[20] and MoE Mamba, which integrates mixture-of-experts with Mamba, requiring 2.2× fewer training steps than standard Mamba^[15]

The continuous-time SSM is defined by linear ordinary differential equations:^[1][2]

x'(t) = Ax(t) + Bu(t)
y(t) = Cx(t) + Du(t)

where:

• x(t) ∈ ℝᴺ is the state vector (N-dimensional latent state)
• u(t) ∈ ℝ is the input signal
• y(t) ∈ ℝ is the output signal
• A ∈ ℝᴺˣᴺ is the state transition or dynamics matrix
• B ∈ ℝᴺˣ¹ is the control matrix
• C ∈ ℝ¹ˣᴺ is the output matrix
• D ∈ ℝ is the feedthrough term

To implement SSMs on digital computers, the continuous system must be discretized. The most common approach uses the zero-order hold (ZOH) method with step size Δ:^[1][21]

x̄ₖ = Āxₖ₋₁ + B̄uₖ
yₖ = Cx̄ₖ + Duₖ

where the discrete parameters are:

Ā = exp(ΔA)
B̄ = (exp(ΔA) - I)A⁻¹B

This discretization highlights two complementary views:^[1]

1. Recurrent view: Efficient O(N) linear-time inference by maintaining state
2. Convolutional view: Parallel O(N log N) training via FFT-based convolutions^[10]

The convolution kernel K̄ can be precomputed as:

K̄ₖ = CĀᵏB̄

This duality allows SSMs to combine the inference efficiency of RNNs with the training parallelism of CNNs and Transformers.^[1]

A fundamental advantage of SSMs is their computational complexity compared to Transformers:^{[2][12][21][22]}

Specifically, Transformers have the following complexity profile:

• Training complexity: O(N²D) where N is sequence length, D is dimension
• Inference complexity: O(N²D) due to attention over all previous tokens
• Memory: O(N) for KV cache, growing linearly with sequence length

In contrast, State Space Models, have the following complexity profile:

• Training complexity: O(N log N) via FFT for convolutional view
• Inference complexity: O(N) linear time per token (only update hidden state)
• Memory: O(1) constant for state, independent of sequence length

This means that SSMs scale better during training and achieve linear-time generation, while Transformers have quadratic complexity that becomes prohibitive for very long sequences.^[3][21] At sequence lengths beyond 8,000-16,000 tokens, SSMs typically become significantly faster than Transformers.^[22]

SSMs have demonstrated strong performance on various NLP tasks, many of which are discussed above in more detail, and include:^[14][23]

• Language modeling competitive with Transformers of similar or larger size
• Long-document understanding with contexts up to 256K tokens
• Strong performance on question answering, summarization, and text classification

The selective mechanism in Mamba has proven particularly effective for discrete modalities like language, addressing early limitations of S4 in this domain.^[14]

Vision applications of SSMs have addressed many common areas of visual processing also tackled by Transformers, but with more of a focus on time series data.^[15][24] Briefly, these include image classification on ImageNet, sequential image tasks (e.g., sequential CIFAR-10), and video understanding and generation. Notably, Vision Mamba (Vim) achieved competitive results with Vision Transformers.^[25]

SSMs excel at audio tasks due to their continuous-time formulation.^[26] Audio applications include speech generation with models like SaShiMi,^[26] extremely efficient audio classification on automatic speech recognition benchmarks,^[27]

The continuous nature of SSMs makes them well-suited for a wide variety of time series problems.^[28] Generally, these include genomic sequence modeling (million-length DNA sequences), ^[29] climate and weather prediction,^[30] and medical time series analysis.^[31]

Despite their advantages, SSMs face several challenges:^[14][32]

1. Associative recall: Pure SSM architectures may struggle with tasks requiring precise retrieval of specific information from long contexts, where attention mechanisms excel
2. Training speed at short sequences: Highly optimized Transformer implementations can be faster than SSMs at sequence lengths below 2,000-4,000 tokens
3. State capacity: Fixed-size hidden states may saturate on extremely long sequences, though this can be mitigated with larger state dimensions
4. Discrete modalities: While Mamba addressed this, earlier SSMs like S4 showed higher perplexity on language tasks compared to Transformers

These limitations have motivated hybrid architectures that combine SSMs with attention mechanisms to leverage the strengths of both approaches.^[17][18] Often, as in many cases above, these are referred to simply as SSM architectures because they are not pure Transformer architectures.

• Recurrent neural network
• Transformer (machine learning model)
• Attention (machine learning)
• Long short-term memory
• Control theory
• Mamba (deep learning architecture)

• Mamba official GitHub repository
• S4 official GitHub repository
• The Annotated S4 tutorial
• HiPPO official code
• AI21 Jamba models