Parts of Speech Emerge as Categories in SAE Latent Space
speech
| Source: HF Papers | Original article
Researchers find that sparse autoencoders reveal part‑of‑speech categories as emergent structures in language‑model latent spaces.
A new EMNLP 2026 paper demonstrates that parts‑of‑speech (PoS) categories emerge as coherent structures within the latent space of Sparse Autoencoders (SAEs) applied to large language models. The study, authored by researchers from the University of Pisa, probes a LLaMA‑3‑8B model equipped with an SAE and extracts activations on the UD English GUM treebank. By treating PoS as a controlled test case, the authors show that PoS distinctions can be recovered with high accuracy from the SAE activations, yet the mapping is not a simple one‑to‑one correspondence between individual latents and grammatical categories. Instead, PoS information appears distributed across structured groups of features.
The findings matter because they advance the interpretability of modern language models. SAEs have been promoted as a promising lens for dissecting internal representations, but the extent to which they capture linguistic structure has remained uncertain. Demonstrating that morpho‑syntactic information is present—albeit in a distributed form—suggests that SAEs can serve as a diagnostic tool for researchers seeking to understand how models encode grammar. It also raises questions about the granularity of probing methods that assume isolated neurons correspond to discrete linguistic concepts.
Future work will likely explore whether similar emergent groupings arise for other linguistic phenomena, such as semantics or discourse relations, and whether the pattern holds across languages and larger model families. Tracking follow‑up studies that extend the analysis to different architectures or that leverage the grouped latent representations for downstream tasks will be essential for gauging the broader impact of this interpretability approach.
Sources
Back to AIPULSEN