Exploring Mixture-of-Experts: From 1991 vs DeepSeek-V3
deepseek google
| Source: Mastodon | Original article
A new article examines the evolution of Mixture of Experts from its 1991 origins to DeepSeek‑V3, as LLM labs grapple with scale versus cost.
DeepSeek has unveiled its latest language model, DeepSeek‑V3, a 671‑billion‑parameter Mixture‑of‑Experts (MoE) system that activates only about 37 billion parameters per token. The model, trained over two months at a reported cost of $5.58 million, demonstrates how MoE architectures can deliver dense‑model performance while keeping compute and expense in check.
The MoE concept is not new – it traces back to the 1991 “adaptive mixtures of local experts” paper that introduced the idea of routing inputs to specialised sub‑models. DeepSeek’s implementation scales the principle to modern LLM sizes, offering a stark contrast to the “scale‑at‑any‑cost” approach that dominates many labs today. By keeping the active parameter count low, DeepSeek‑V3 can generate text with the capacity of a much larger dense model but at a fraction of the inference cost.
This development matters because the AI community is wrestling with the trade‑off between ever‑larger models and the soaring energy and financial bills they entail. A cheaper, high‑capacity MoE could democratise access to cutting‑edge LLMs, lower barriers for startups, and reshape pricing structures for cloud‑based inference services. It also aligns with our recent coverage of DeepSeek’s rise – from becoming the default model on ChatGPT Codex to the impressive 23 seconds/token performance of DeepSeek v4.1 flash on modest hardware.
Looking ahead, the next wave of DeepSeek MoE models – the V4 family (1.6 trillion‑parameter “Pro” and 284 billion‑parameter “Flash”) – is already slated to hit the market. Observers will watch how these larger MoE variants influence competitive dynamics, especially against Google’s own MoE research, and whether they trigger broader adoption of sparse‑activation architectures across the industry.
Sources
Back to AIPULSEN