HN Show: Maple – Sub-2-BPW Moe Pruning to Run Gemma 4 26B at 45 TPS on 6GB GPU
gemma gpu
| Source: HN | Original article
Maple uses sub‑2‑BPW MoE pruning to run the 26‑billion‑parameter Gemma‑4 model at 45 tokens per second on a 6 GB GPU.
A new Show HN post introduces “Maple,” a pruning technique that pushes the limits of what can be run on consumer‑grade graphics hardware. By applying sub‑2‑bit‑per‑weight (BPW) Mixture‑of‑Experts (MoE) pruning, the authors claim the 26‑billion‑parameter Gemma 4 model can generate text at roughly 45 tokens per second on a GPU with just 6 GB of memory.
The breakthrough matters because it narrows the gap between massive, cloud‑hosted language models and the modest compute resources available to researchers, startups, and hobbyists. Running a 26 B model on a single low‑end GPU could slash inference costs, lower the barrier to experimentation, and reduce the environmental footprint of AI workloads. The result also echoes earlier work we covered on 2026‑11‑08, where a 31 B video‑audio diffusion model was compressed to 14 B parameters using INT8 quantisation, underscoring a growing trend toward aggressive model size reduction without sacrificing speed.
What to watch next is whether Maple’s approach scales to other large models and how it performs on real‑world tasks beyond raw token throughput. The AI community will be looking for independent benchmarks, integration into popular inference frameworks, and any trade‑offs in accuracy or stability. If the method proves robust, it could accelerate the adoption of high‑capacity models in edge devices and small‑scale deployments, reshaping the economics of AI services and prompting hardware vendors to reconsider the memory requirements of future GPUs.
Sources
Back to AIPULSEN