Gemini 4 and Gemma 4 Put Cost Ahead of Complex Math
gemini gemma
| Source: Dev.to | Original article
Gemini 4 and Gemma 4, together with five other AI models, solved all 15 AIME 2025 I problems at both low and high thinking settings; the high setting cost up to 4.8× more while performance gains were not statistically significant.
Seven AI models were put through a head‑to‑head test on the fifteen problems from the 2025 American Invitational Mathematics Examination (AIME) I, each run twice through the Gemini API: once at the model’s lowest “thinking” setting and once at its highest, while keeping a strict cost ceiling. The lineup included Gemma 4 (26 billion‑parameter and 31 billion‑parameter variants), Gemini 3.5 Flash‑Lite, Gemini 3.8 Flash, and the three Gemini 4 configurations dubbed Hydrogen, Carbon and Argon.
The experiment showed that cranking the thinking setting up raised the price tag by a factor of 2.6‑4.8 across every paid model, yet the performance gain was modest. Hydrogen, Carbon and Argon solved all fifteen problems at both settings, and the difference in scores between low and high settings was never large enough to rule out random variation. In other words, the extra compute did not translate into a decisive accuracy edge on this hard‑math benchmark.
Why it matters is twofold. First, it provides the first systematic, cost‑aware comparison of “low‑thinking” versus “high‑thinking” modes on a truly challenging academic test, giving developers a concrete sense of the trade‑off between token‑price and marginal gains. Second, the fact that the top‑tier Gemini 4 variants already hit a perfect score at the cheapest setting suggests that, for certain high‑stakes tasks, users may not need to pay premium rates for higher‑setting runs.
Looking ahead, the AI community will be watching for broader evaluations that extend beyond AIME to other domains such as scientific reasoning or coding. Further, any adjustments to Google’s Gemini pricing—especially for the high‑thinking tier—could shift the cost‑benefit calculus. Finally, follow‑up studies may explore whether newer model releases or fine‑tuned variants can break the current ceiling of performance without inflating expenses.
Sources
Back to AIPULSEN