Gated DeltaNet adopts §3‑bit quantization as NVFP4 applies W4A4 to the recurrent half of the hybrid 27B LLM
huggingface qwen
| Source: HF Papers | Original article
Gated DeltaNet in hybrid 27B LLMs survives 4‑bit quantization, enabling NVFP4 W4A4 for the recurrent half while earlier attempts kept GDN at 8‑ or 16‑bit.
A new quantization recipe shows that the recurrent half of Qwen 3.8‑27B’s hybrid architecture can survive full 4‑bit compression. The model blends conventional softmax attention with linear‑attention blocks built from Gated DeltaNet (GDN), a layer that keeps a fixed‑size recurrent state instead of a growing key‑value cache. Early community attempts at 4‑bit quantization left the 48 GDN layers in 8‑ or 16‑bit precision, fearing that aggressive compression would break the recurrent summariser.
The breakthrough comes from applying NVIDIA’s FP4 format (NVFP4) with a W4A4 scheme to the entire model except for a handful of components that remain in bf16. According to a Hugging Face repository, the vision tower, language‑model head, DeltaNet conv1d and the MTP head are kept in bf16, while everything else—including the GDN recurrent state—is stored in NVFP4 W4A4. The result is a fully 4‑bit model that still runs speculative decoding out of the box.
Why it matters is twofold. First, the memory footprint drops dramatically, allowing the 27‑billion‑parameter hybrid LLM to run on a single GPU with far less VRAM than previously required. Second, the speed gains reported by Unsloth’s NVFP4 benchmarks suggest that the model retains its strong performance on agentic coding, vision, and chat tasks, even when compressed. This aligns with the recent Cerebras deployment of Qwen 3.8‑27B, where the model already demonstrated 1 500 tokens / s throughput.
What to watch next are the downstream integrations. The sgl‑project’s recent NVFP4 MoE work, Ollama’s support for an “nvfp4” tag, and Unsloth’s desktop client all point to rapid adoption across Linux, Windows and Apple Silicon environments. Further benchmarking will reveal whether the 4‑bit GDN can match the quality of higher‑precision runs, and whether other hybrid LLMs will follow suit. If the trend holds, 4‑bit quantization could become the default path for deploying large, recurrent‑state models on commodity hardware.
Sources
Back to AIPULSEN