Study Finds Transformers Can Simultaneously Encode Two Thoughts via Linear Superposition in LLMs
| Source: HF Papers | Original article
Researchers demonstrate that large language models can linearly combine inputs from separate text streams, outputting a superposition of next‑token distributions, revealing underlying linearity.
A new study by Pavel Tikhonov, Anton Korznikov and Matvey Mikhalchuk provides the first empirical evidence that large language models (LLMs) can process two distinct text streams simultaneously. The researchers demonstrate that, despite the heavily non‑linear architecture of Transformers, the models exhibit a fundamental linearity: when the embeddings of two separate inputs are combined linearly, the resulting next‑token distribution is a superposition of the distributions each input would generate on its own. The authors label this behavior the **Superposition Linearity Hypothesis**.
The finding matters because it challenges the prevailing view of LLMs as purely non‑linear black boxes. If a model can hold “two thoughts” at once, it suggests that internal representations are more composable than previously thought. This could open new avenues for probing model internals, designing multi‑task prompts that deliberately blend tasks, and even simplifying certain inference pipelines by exploiting linear combinations of inputs. Moreover, the result hints at a deeper connection between the Transformer’s attention mechanisms and linear algebraic structures, potentially informing future architecture tweaks aimed at efficiency or interpretability.
The next steps will likely involve testing the hypothesis across a broader range of model sizes and training regimes, as well as exploring practical applications such as dual‑task generation or controlled output blending. Researchers will also watch for follow‑up work that quantifies the limits of superposition—how many streams can be combined before the signal degrades—and whether the effect can be harnessed without sacrificing accuracy. If the phenomenon proves robust, it could reshape how developers think about prompting and fine‑tuning large language models.
Sources
Back to AIPULSEN