Factored One‑Sided Adam Moment Enhances Practical, Scalable SOAP
training
| Source: HF Papers | Original article
A new factored one-sided Adam-moment method seeks to make the SOAP second-order optimizer more practical and scalable by cutting its memory and computational costs.
A new variant of the SOAP optimizer has been unveiled, promising to make second‑order training methods practical at scale. The technique, dubbed **Factored One‑sided Adam‑Moment**, modifies SOAP’s internal state handling so that the costly eigendecomposition and memory overhead that have limited its adoption are dramatically reduced.
Second‑order optimizers such as Shampoo and the original SOAP have been praised for their ability to converge faster than the ubiquitous AdamW in large‑scale transformer training. Yet, their practical use has been hampered by the need to store full‑rank optimizer state and to perform expensive matrix operations each step. By factoring the Adam‑style moment estimates and applying them only on one side of the curvature matrix, the new approach retains SOAP’s convergence advantages while cutting the per‑step memory footprint and computational load.
The development matters because it lowers a key barrier to deploying more sophisticated optimizers in production‑grade models. Faster convergence translates into fewer training epochs, which can shave weeks off the development cycle of massive language models and reduce cloud‑compute costs. If the method lives up to its promise, it could shift the optimizer landscape away from first‑order methods that dominate today, especially in environments where hardware memory is at a premium.
The next steps will be watching for benchmark releases that compare Factored One‑sided Adam‑Moment‑enhanced SOAP against AdamW and the original SOAP on standard transformer workloads. Open‑source integration into popular deep‑learning frameworks would also signal broader adoption. Finally, follow‑up research may explore extending the factorization idea to other second‑order schemes, potentially opening a new class of scalable, memory‑efficient optimizers for the next generation of AI models.
Sources
Back to AIPULSEN