PersonaCUABench Benchmarks Computer-Use Agents Across Multiple Personas
agents benchmarks
| Source: HF Papers | Original article
Researchers unveil PersonaCUABench, a new benchmark that evaluates computer-use agents on their ability to serve diverse user personas and handle ambiguous or constrained instructions.
A new benchmark called **PersonaCUABench** has been released to probe how well computer‑use agents (CUAs) cope with the nuances of real‑world users. Unlike prior evaluations that present a fixed task description, PersonaCUABench keeps the underlying task constant—74 tasks spanning 11 domains—but swaps the “person” who issues the request. Each of 44 personas, grouped into four categories, rewrites the original instruction in its own words and interacts with the CUA, while the agent only sees the dialogue, never the underlying profile.
The results are stark. Across 3,330 runs on nine leading CUA backbones, average task completion fell from 0.689 to 0.441—a 36 % drop—when the same task was framed by different personas. User‑satisfaction scores slipped from 84.1 % to 62.7 %, and 11.2 % of failures stemmed from users explicitly stopping the agent, a scenario the benchmark’s checker flags as a miss. The performance gap emerges after the agent has already responded to user‑raised constraints, underscoring that merely completing a prescribed instruction does not guarantee serving the requester’s intent.
The benchmark matters because it surfaces a blind spot in current CUA development: agents are typically tuned to static specifications, yet everyday users frequently omit requirements, add ad‑hoc constraints, or demand approval before consequential actions. Ignoring these dynamics can lead to agents that appear competent in lab tests but falter in practice, potentially eroding trust and limiting adoption.
Going forward, researchers will likely explore persona‑aware prompting, dynamic constraint handling, and richer feedback loops to close the gap highlighted by PersonaCUABench. The work dovetails with recent tool‑aware evaluation efforts such as TauIndianBankBench and TAO, suggesting a broader shift toward more human‑centric assessment of autonomous agents. Monitoring how major AI labs adapt their models to these findings will be key to gauging progress in truly user‑responsive AI.
Sources
Back to AIPULSEN