PAR Proposes Progress-Aligned Recurrence for Better Length Generalization in State Tracking
| Source: HF Papers | Original article
A new method, Progress‑Aligned Recurrence (PAR), improves length generalization in state‑tracking tasks by aligning looped language‑model iterations with progress, overcoming limits of final‑answer supervision.
Ant International and researchers from the National University of Singapore have unveiled a new mechanism called Progress‑Aligned Recurrence (PAR) to tackle the long‑standing problem of length generalisation in sequential state‑tracking tasks. The approach, described in a paper released on October 20, gates each recurrent update by the amount of task progress rather than by a fixed token count. By aligning the prediction at iteration k with the state after the first k operations and normalising recurrent states to the norm of the first iteration, PAR prevents the scale drift that typically plagues models trained only with final‑answer supervision.
The authors demonstrate the impact on a benchmark called swap‑S5, where a Qwen2.5‑0.5 B model trained on sequences of at most 32 moves reaches 95.5 % mean full‑state accuracy when evaluated on 512‑move sequences—fourteen times longer than seen during training. By contrast, the same model without PAR stalls at 55.7 % under the same conditions. The improvement is achieved without altering the underlying architecture, simply by changing how recurrent updates are timed and normalised.
Why it matters is twofold. First, many real‑world applications—rule‑table reasoning, object tracking, and other step‑by‑step decision processes—require models to extrapolate far beyond the lengths encountered during training. Second, the finding shows that looped language models can be made to use additional iterations effectively, addressing a key limitation of current recurrent and linear‑attention designs.
The next steps will likely involve scaling PAR to larger models and testing it on diverse domains such as program synthesis or long‑form planning. Observers will watch for follow‑up benchmarks, integration into mainstream frameworks, and whether the technique can bridge the gap between transformer‑style depth and recurrent‑style flexibility in production AI systems.
Sources
Back to AIPULSEN