Sparse Autoencoders Shift Representation Allocation
| Source: HF Papers | Original article
Study shows prompt transformations can redistribute activity across sparse autoencoder features, so a feature change seen on one prompt may not recur on others.
A new study introduces the Redistribution Sensitivity Score (RSS), a metric designed to spot and quantify how sparse autoencoders (SAEs) reshuffle activity across their learned features when prompts are altered. SAEs break down language‑model activations into a compact set of sparse features, a technique that has become central to mechanistic interpretability work. The researchers show that even transformations of a prompt that leave the underlying request unchanged can shift activation weight from one feature to another. Crucially, a change observed for one transformed prompt does not necessarily repeat for other variants, and a recurring shift does not automatically translate into a downstream computation error.
Why this matters is twofold. First, the finding challenges the assumption that stable feature attribution guarantees robust downstream performance; subtle redistribution can undermine readouts without obvious sign‑posts. Second, RSS offers a practical tool for model auditing, allowing researchers to isolate “compact supports” – small groups of features whose movement explains substantial changes in downstream task outcomes. This adds a layer of diagnostic precision to the growing toolbox for probing large language models.
The work follows our earlier coverage of sparse‑autoencoder behavior, notably the Oct 17 article on behavioral compositionality, and pushes the conversation toward robustness. Going forward, the community will watch for extensions of RSS to larger, multimodal models and for integration into interpretability platforms. If RSS proves effective at flagging hidden failure modes, it could become a standard checkpoint in the development and deployment of AI systems that rely on sparse representations.
Sources
Back to AIPULSEN