Moonshot AI Unveils Hybrid Attention Architecture That Outperforms Full Attention
| Source: Mastodon | Original article
Moonshot AI developed a hybrid attention architecture called Kimi Linear that outperforms full‑attention models.
Moonshot AI’s Kimi team has unveiled a new attention mechanism that, according to a technical report posted on arXiv in October 2025 and now gaining traction on Hacker News, outperforms the traditional full‑attention Transformer while using far fewer resources. The architecture, dubbed **Kimi Linear**, combines a linear‑attention core – called Kimi Delta Attention (KDA) – with a gated‑per‑channel design derived from Gated DeltaNet. In head‑to‑head tests across short‑context, long‑context and reinforcement‑learning scaling regimes, Kimi Linear delivers higher quality on reasoning, retrieval and other tasks, while cutting the key‑value cache by roughly 75 % and accelerating decoding by more than six times at a one‑million‑token window.
The breakthrough matters because the quadratic memory and compute cost of standard attention has become a bottleneck as LLMs push toward ever‑larger context windows. KV‑cache footprints that swell to tens of gigabytes slow inference and inflate hardware requirements, limiting both research experimentation and commercial deployment. By showing that a hybrid linear approach can retain – and even improve – model performance, Moonshot AI challenges the long‑standing assumption that full attention is the only path to high‑quality results.
The next steps will reveal how quickly the community adopts Kimi Linear. Watch for open‑source releases of the KDA kernel, integration into popular model‑training frameworks, and independent benchmark suites that verify the reported gains. Industry players may also explore the architecture for cost‑sensitive inference services, while academic groups are likely to probe the theoretical limits of per‑channel gating and hybrid attention. If the early results hold, Kimi Linear could reshape the efficiency landscape for next‑generation large language models.
Sources
Back to AIPULSEN