RealSWE Evaluates Coding Agents with Realistic User Requests
agents benchmarks
| Source: HF Papers | Original article
Researchers introduce RealSWE, a new benchmark that assesses coding agents on concise, unstructured user requests, exposing a gap with the longer, curated tasks of SWE‑bench.
RealSWE, a new benchmark that measures coding agents against the sort of terse, unstructured prompts developers actually type, has revealed a notable performance gap in today’s leading large‑language models. Researchers evaluated seven contemporary LLM‑based coding agents on RealSWE’s six‑category suite of realistic user requests and found that resolution rates fell by an average of 6.4 percentage points compared with the traditional SWE‑bench style tasks, which are built from curated, information‑rich GitHub issues.
The finding matters because most public evaluations still rely on SWE‑bench‑type benchmarks, giving a skewed picture of how well agents will fare in everyday development workflows. By exposing the drop in success when models confront the brevity and ambiguity of real‑world prompts, RealSWE challenges providers to rethink training and prompting strategies before they are rolled out in IDE extensions, terminal assistants, or CI pipelines.
The study arrives as vendors are already positioning their tools for realistic use cases. Zhipu’s GLM‑5.3, for example, claims a 50 percent improvement on the private Z.ai Code Bench, which also emphasizes complex local environments. Anthropic’s Claude Code advertises deep code‑base awareness and file‑editing capabilities, while open‑source projects such as OpenDesign let users plug in their own agents for end‑to‑end file generation.
What to watch next is whether model developers will tune their systems specifically for RealSWE‑style inputs and how benchmark results will influence product claims and pricing tiers. Industry observers will also be looking for follow‑up studies that expand the six request categories or integrate RealSWE into broader evaluation suites such as DeepSWE and Terminal‑Bench. The shift toward realistic testing could become a new yardstick for judging the true productivity gains of AI‑driven coding assistants.
Sources
Back to AIPULSEN