Rethinking Critic Learning in PPO to Tackle Value Flattening
reinforcement-learning
| Source: HF Papers | Original article
Researchers reveal a systematic failure called Value Flattening in PPO critics for reinforcement learning of large language models and outline strategies to understand and mitigate it.
A new study has identified a previously overlooked flaw in the way Proximal Policy Optimization (PPO) trains its critic when aligning large language models (LLMs). The researchers observed that critics often suffer from “Value Flattening,” a systematic failure where state‑value estimates—derived from multiple Monte‑Carlo continuations—shift abruptly across intermediate states instead of varying smoothly. This distortion reduces the usefulness of the critic’s variance‑reduction role and can stall policy improvement during reinforcement‑learning‑from‑human‑feedback (RLHF) runs.
The authors demonstrate that the flattening effect stems from the dense supervision typically applied to every token in a response, which can drown out informative signal in sparsely rewarded regions. To counteract the problem they introduce SP³O (Sparse PPO), a lightweight modification that supplies supervision only to a small, well‑spaced subset of states per generated answer. Experiments show that this sparse supervision restores a more faithful value landscape, leading to faster convergence and higher performance on alignment and reasoning benchmarks.
The finding matters because PPO has become the de‑facto method for LLM alignment, and any hidden weakness in its training loop can ripple through commercial and research deployments that rely on safe, reliable model behavior. By exposing Value Flattening, the work prompts a re‑examination of critic design across the field and offers a practical fix that can be adopted without major architectural changes.
Going forward, the community will watch for broader validation of SP³O on diverse model families and tasks, as well as for integration into popular RLHF toolkits. If the approach scales, it could become a new standard for critic training, tightening the feedback loop that underpins the next generation of aligned AI systems.
Sources
Back to AIPULSEN