Looped Transformers Scale Better with Truncated Backpropagation, Study Finds
bias training
| Source: HF Papers | Original article
Researchers evaluate the impact of truncating backpropagation through time on looped transformers, which reuse a shared layer block to boost depth without extra parameters.
A new scaling study has examined whether truncating back‑propagation through time (BPTT) is a worthwhile shortcut for “looped” transformers – models that recycle a single block of layers across multiple forward passes to deepen computation without adding trainable parameters.
The researchers trained almost 600 variants, systematically varying model size, token budget, forward‑loop count and the depth of BPTT. By fitting a single scaling law that captures the trade‑off between compute, loop count and gradient fidelity, they found that truncating BPTT only pays off when the number of forward loops is fixed in advance. In all other cases, running fewer loops with full BPTT yields better performance for the same compute budget.
The findings matter because looped transformers have been promoted as a low‑cost route to the deeper networks that drive recent language‑model breakthroughs. Truncating BPTT can cut training time, but the study shows it introduces biased gradients that erode the very advantage looped designs aim to provide. The derived scaling law gives practitioners a concrete rule‑of‑thumb for allocating compute: either keep the loop count constant and truncate, or otherwise abandon truncation and let the model train through fewer, fully‑back‑propagated loops.
Going forward, the community will watch how these guidelines shape large‑scale pre‑training pipelines, especially as firms seek to stretch limited GPU budgets. Follow‑up work may explore hybrid strategies—dynamic loop counts, adaptive truncation, or alternative gradient‑estimation techniques—to retain the compute savings while mitigating bias. If the scaling law holds across domains, it could become a standard reference for efficient transformer design, influencing both research labs and commercial AI developers.
Sources
Back to AIPULSEN