FLAMIA: FLAsh MIcroscaling launches fast, causally safe training on Blackwell
nvidia training
| Source: HF Papers | Original article
Researchers introduce FLAMIA, a flash microscaling attention method that speeds up Transformer training on NVIDIA Blackwell GPUs while preserving causal safety.
Yandex has unveiled FLAMIA, a new family of attention kernels designed to exploit the low‑precision matrix‑multiplication capabilities of NVIDIA’s Blackwell GPUs. The method, detailed in a paper released this week, combines MXFP8 and mixed NVFP4/FP8 training paths with a token‑block scaling technique that reuses quantized operands across matrix orientations. Crucially, FLAMIA incorporates safeguards that prevent scale‑induced leakage of causal information, ensuring that forward and backward passes remain consistent.
Attention remains the dominant cost in training Transformer models, especially when dealing with long context windows. While Blackwell GPUs promise high‑throughput, low‑precision computation, previous attempts to apply such precision to attention have been hampered by quantization overhead and the risk of causal information leakage. FLAMIA’s design directly addresses these challenges, delivering a reported 1.4× speed‑up in training without compromising the causal safety of the model.
The breakthrough matters because faster, cheaper training lowers the barrier to scaling large language models and could accelerate research into long‑context applications. By keeping the attention computation both efficient and secure, FLAMIA may also influence how future hardware‑software co‑designs are approached, encouraging broader adoption of sub‑8‑bit arithmetic in critical model components.
The next steps to watch include whether FLAMIA will be integrated into popular deep‑learning frameworks, how other organisations benchmark its performance on Blackwell‑based clusters, and whether the technique can be adapted to upcoming GPU architectures. If the early results hold up, FLAMIA could become a standard tool for cost‑effective, high‑throughput Transformer training across the AI industry.
Sources
Back to AIPULSEN