Frequency-Aware Remasking Improves Test-Time Scaling of Diffusion Language Models
inference
| Source: HF Papers | Original article
Researchers propose a test‑time scaling method for diffusion language models that uses frequency‑aware remasking to improve token selection during iterative denoising.
A new paper released this week proposes a two‑stage test‑time scaling technique for diffusion language models (DLMs) that promises higher accuracy without a proportional increase in compute. The authors frame scaling as an adaptive allocation of both inference resources and decoding strategies. First, a fast base sampler tackles “easy” queries, freeing up capacity for the more challenging cases. Those hard questions are then processed with a frequency‑aware remasking step that monitors how often token predictions change across denoising steps and keeps frequently‑flipping tokens in play for further refinement.
The approach directly addresses a known weakness of existing certainty‑aware methods, which rely on instantaneous confidence or entropy and can be misled on difficult prompts. By using a trajectory‑level signal—how often a token’s prediction varies—the frequency‑aware remasking provides a richer cue for where additional computation is needed.
Experimental results on the LLaDA‑8B‑Instruct and Dream‑7B‑Instruct models show measurable gains on the MATH‑500 and HumanEval benchmarks, including the recovery of several problems that the base sampler missed entirely. The findings suggest that DLMs can achieve better performance with a modest, targeted increase in inference effort, a step toward making diffusion‑based text generation more practical for real‑world applications.
The next questions are whether the method scales to larger, commercial‑grade models and how it performs on a broader suite of tasks such as code generation, dialogue, or long‑form writing. Follow‑up work is likely to explore integration with existing inference pipelines, automated detection of “hard” queries, and potential extensions of frequency‑aware signals to other generative architectures.
Sources
Back to AIPULSEN