Next-Chunk Reasoning RL vs SFT: Is It Better? Rethinking Training Without CoT Data
reasoning training
| Source: HF Papers | Original article
Researchers compare next-chunk reasoning RL to supervised fine-tuning for training on non‑CoT data like worked solutions and textbook derivations.
A new study from Yinhao Tang and ten co‑authors challenges the growing belief that reinforcement‑learning (RL) finetuning is the superior way to teach large language models to reason. The paper, titled *Is Next‑Chunk Reasoning RL Really Better than SFT?*, evaluates “next‑chunk reasoning RL” – an approach that extracts implicit reasoning traces from corpora that contain rich derivations but no explicit chain‑of‑thought (CoT) annotations – against a simpler “mixed supervised fine‑tuning” (Mixed SFT) regime.
The authors report that Mixed SFT not only reaches a higher performance ceiling after the RL‑based verification step (post‑RLVR) but does so with dramatically lower computational cost, requiring more than 60 × less training compute than the RL alternative. The result suggests that, at least for “no‑CoT” data such as worked‑out textbook solutions, the added complexity of RL‑driven reasoning does not translate into better outcomes.
Why it matters: RL‑based finetuning has been promoted as the next frontier for improving reasoning in large models, especially after earlier reports that RL helped multimodal models see better or that aggregating multiple answers outperforms single‑shot outputs. If a straightforward supervised mix can outperform RL while slashing compute, research labs and commercial teams may rethink their training pipelines, reallocating resources toward data curation and efficient SFT rather than expensive RL loops. The finding also tempers expectations that RL alone can bridge the gap to human‑level reasoning on tasks lacking explicit CoT signals.
What to watch next: The community will likely probe whether the Mixed SFT advantage holds across other domains, larger model scales, and different no‑CoT datasets. Follow‑up work may explore hybrid schemes that combine the cheap supervision of Mixed SFT with targeted RL refinements, or investigate how verification stages (RLVR) can be made more effective without the heavy RL pre‑training cost. The debate over the optimal recipe for reasoning‑rich LLMs is set to intensify as more groups test these claims on real‑world workloads.
Sources
Back to AIPULSEN