SMELT Publishes Scaling Laws for Compute‑Matched MoE Looped Transformers
embeddings
| Source: HF Papers | Original article
Researchers study looping a shared block in Mixture‑of‑Experts Transformers while matching per‑token FLOPs, showing earlier size‑fixed tests may mix architectural gains with extra compute.
A new arXiv paper titled **“SMELT: Scaling Laws for Compute‑Matched MoE Looped Transformers”** proposes a concrete recipe for making mixture‑of‑experts (MoE) transformers more compute‑efficient by looping a shared block of layers. The authors call the method SMELT – Sparse MoE Transformer, middle layers Loop Twice – and demonstrate that looping the middle 50 % of MoE layers while narrowing the hidden dimension, raising the expert count, scaling the looped residuals by ½, and using smaller attention heads with a higher GQA ratio can cut per‑token FLOPs by roughly 6.8 %–18 % without sacrificing model capacity.
The study scales the approach across four model sizes up to 54 billion non‑embedding parameters, fitting a separate Chinchilla‑style scaling law for each architecture. Results show that MoE models equipped with the SMELT recipe consistently outperform dense transformers when matched for compute, and that the efficiency gap widens as models grow larger and training budgets increase. By keeping the KV cache size nearly unchanged across extra layer executions, the method preserves inference latency while delivering deeper effective computation through repeated execution of the same block.
Why it matters is twofold. First, it isolates the genuine architectural benefit of looping from the confounding effect of extra FLOPs that has plagued prior evaluations of looped transformers. Second, the derived scaling laws give researchers and engineers a principled way to choose model depth, expert count and hidden size for a given computational budget, potentially reshaping how large‑scale language models are built and trained.
The community should watch for open‑source releases of the SMELT code, benchmark suites that compare looped MoE models against the latest dense alternatives, and any follow‑up work that integrates the recipe into production‑grade frameworks. If the efficiency gains hold at scale, SMELT could become a standard design pattern for next‑generation AI systems seeking to stretch compute further without proportional cost increases.
Sources
Back to AIPULSEN