EOS Token Mismatch Triggers Length Inflation in On-Policy Distillation
llama qwen
| Source: HF Papers | Original article
A new study has pinpointed the cause of “length inflation” in on‑policy distillation (OPD), a technique used to transfer knowledge from larger teacher language models to smaller student models. Researchers observed that, during OPD, student responses can swell far beyond expected lengths, sometimes exhausting the allotted generation budget. The analysis traces the problem to a mismatch in termination tokens: the base student model and the post‑trained teacher disagree on which token signals the end of a sequence. This misalignment forces the student to keep generating tokens until a teacher‑defined stop condition is met, inflating output length.
The finding matters because OPD is increasingly popular for fine‑tuning large language models efficiently. Length inflation not only wastes compute resources but also risks degrading user experience and complicates downstream deployment where strict token limits are common. By highlighting termination‑token mismatch as a key driver, the work offers a concrete target for mitigation, complementing broader efforts to stabilise OPD training and improve token‑level teachability.
The authors suggest that future research should explore stabilization strategies that reconcile teacher‑student termination signals, as well as metrics that better predict token teachability beyond raw KL divergence. Monitoring how these solutions are integrated into mainstream model‑distillation pipelines will be essential, especially as the community seeks to scale OPD without sacrificing efficiency or safety. The study adds to a growing body of work dissecting OPD dynamics and sets the stage for more robust, budget‑aware distillation methods.
Sources
Back to AIPULSEN