ICML 2026 · International Conference on Machine Learning

Kalman Linear Attention: Parallel Bayesian Filtering for Efficient Language Modelling and State Tracking

“Towards probabilistic LLMs - one Bayesian layer at a time”

Vaisakh Shaj · Cameron Barker · Aidan Scannell · András Szécsényi · Elliot J. Crowley · Amos Storkey

University of Edinburgh, United Kingdom

Lay summary

The AI systems behind today's chatbots, called transformers, are powerful but grow slow and resource-hungry as the text they handle gets longer. Researchers have built leaner, faster alternatives, but these often struggle to keep careful track of information across long passages - a skill that genuine reasoning needs.

We turned to an unlikely source: the Kalman filter, a 60-year-old algorithm that estimates the hidden state of a system from noisy measurements while keeping track of its own uncertainty. It famously helped guide the Apollo missions to the Moon and remains a gold standard in engineering and neuroscience. The catch was that it seemed to work only one step at a time - far too slowly to learn from the vast text collections modern AI needs. We found that rewriting the filter in a different mathematical form reveals hidden structure that lets all its steps be computed at once, in parallel, and built this into a neural network layer we call Kalman Linear Attention.

Because the model continually reasons about its own uncertainty, useful behaviours that other systems bolt on by hand emerge on their own. It matches or improves on leading efficient models for language, and is among the first such probabilistic methods trained at large scale - a step toward AI that has a built-in sense of what it doesn't know.

Abstract

click to expand

State-space language models such as Mamba and gated linear attention (GLA) offer linear-complexity, parallelisable alternatives to transformers, but often lack the expressivity and robust state-tracking needed for complex reasoning. We address this by reframing sequence modeling through a probabilistic lens, with Bayesian filters as the core primitive. Classical Kalman filters provide principled state and uncertainty estimation but are typically viewed as inherently sequential. We show that reparameterising the Kalman filter in information form casts its updates as an associative scan, enabling efficient parallel training. The resulting Kalman Linear Attention (KLA) layer performs time-parallel probabilistic inference while maintaining explicit belief-state uncertainty, offering strictly more expressive nonlinear updates and gating than GLA variants while retaining their computational advantages. On language modeling tasks, KLA matches or outperforms modern SSMs and GLAs on discrete token-manipulation, state-tracking, and zero-shot commonsense-reasoning benchmarks, and is among the first stacked Bayesian-filtering primitives pretrained at the billion-token scale.

Highlights

Experiments

Why should I use a Bayesian Filter for Language Modelling?

Interpretable in-context learning through explicit uncertainty neurons

Posterior variance of every state dimension over a Selective Copy sequence, with the mean and one standard deviation plotted below; variance spikes at the tokens to be copied and the belief sharpens as more informative tokens arrive.

While solving Selective Copying - the synthetic recall task popularised by Mamba (Gu & Dao, 2023) - KLA additionally gives us interpretable posterior uncertainty by construction, exposed as explicit uncertainty neurons. Clear patterns emerge: uncertainty spikes exactly when a useful “gold” token to be copied arrives, and in general as the context sees more informative tokens the belief sharpens.

Non-linear + parallel recurrence = state-tracking models that scale

The A5 task: five objects A to E in five slots are shuffled by a sequence of swap actions a1, a2, a3; the model must report the arrangement after each action.

A5 is a proxy for state tracking, popularised by Merrill et al. (2024). Intuitively it is a world-modelling problem in miniature: the world state is an arrangement of five objects, every action is a swap, and the task is to predict the world state after k actions have been applied.

Minimum number of blocks required to reach 90 percent accuracy on A5 versus sequence length: Mamba and Transformer grow to four blocks, Kalman Linear Attention stays at two, an RNN stays at one.

Sequence length counts the number of swaps (“actions”), so the task gets harder as the horizon grows. KLA solves it at a constant depth of 1-2 blocks, whereas popular sequence models need depth that grows in proportion to the difficulty.

Generative pretrained KLA is a competitive probabilistic LLM

Zero-shot accuracy table at 45M and 180M parameters across LAMBADA, HellaSwag, PIQA, ARC-E, ARC-C, WinoGrande, OBQA and BoolQ for GPT, Mamba, GDN, KLA and the GPT plus KLA hybrid; GPT plus KLA has the best average at both sizes (38.33 and 44.95).

We stacked KLA blocks up to 24 deep - training stayed stable - and pretrained from scratch on FineWeb-Edu at 45M and 180M parameters (up to ~11B tokens). As far as we know, no stacked Bayesian filter has been trained at this scale before. A standalone KLA pretrained LM matches or beats softmax attention, Mamba and GDN on zero-shot commonsense benchmarks.

Average downstream accuracy versus model size for pure GPT, GPT plus Mamba, GPT plus GDN and GPT plus KLA; GPT plus KLA is highest at 45M (38.33) and 180M (44.95).

Finally, we set up a minimal hybrid LM (a quadratic + subquadratic mix): a GPT trained from scratch, but with its last layer swapped for a single KLA layer. That one-layer addition lifts zero-shot accuracy across benchmarks - and by more than the equivalent Mamba or GDN layer. KLA complements attention rather than merely replacing it.

Citation

@inproceedings{shaj2026kalman,
  title   = {Kalman Linear Attention: Parallel Bayesian Filtering for
             Efficient Language Modelling and State Tracking},
  author  = {Shaj, Vaisakh and Barker, Cameron and Scannell, Aidan and
             Sz{\'e}cs{\'e}nyi, Andr{\'a}s and Crowley, Elliot J. and Storkey, Amos},
  booktitle = {International Conference on Machine Learning (ICML)},
  year    = {2026}
}