SpireBench Tests Long-Term Strategic Skills of Agentic Models Using Slay the Spire
agents benchmarks reasoning
| Source: ArXiv | Original article
Researchers introduce SpireBench, a new benchmark that assesses the long‑horizon strategic abilities of agentic large language models using the game Slay the Spire.
A new open‑source benchmark called **SpireBench** has been released to probe the long‑horizon strategic abilities of agentic large language models (LLMs). Built on the “Slay the Spire 2” video game and the HermesBridge file‑IPC mod, the benchmark records each full run—from the opening Neow decision to death or victory—as a line‑by‑line command log and a structured statistics file. The first trial, dubbed trial‑v0, freezes the evaluation protocol in a “SpireBench‑Lite” contract that strips the original SKILL.md rules down to an evaluation‑focused specification.
The authors tested five frontier LLMs—GPT‑5.5, Claude Opus 4.7, DeepSeek V4 Pro, Gemini 3.1 Pro Preview and GLM‑5.1—across all five playable characters at Ascension 0, running 25 zero‑shot games in total. Despite matching human‑level performance on short‑term tasks, every model ultimately “died” before achieving victory, exposing a consistent weakness in long‑term risk management and strategic consistency.
Why it matters: most existing agentic benchmarks are short, episodic challenges that do not stress the planning depth required for real‑world decision making. By forcing models to navigate a multi‑stage, branching game where early choices echo through dozens of turns, SpireBench highlights a gap between raw language capability and sustained strategic reasoning. The findings echo concerns raised in our earlier coverage of game‑agent competitions, where heuristic learning showed limits in complex environments.
What to watch next: the SpireBench team plans to iterate on the protocol, expand character and difficulty settings, and open the benchmark to community contributions. Researchers will likely use the dataset to fine‑tune or redesign model architectures aimed at better long‑range planning. As the field pushes toward more autonomous AI agents, the community’s response to SpireBench will be a barometer for progress in strategic AI safety and reliability.
Sources
Back to AIPULSEN