Work2Skill taps human GUI experience to build reusable agent skills
agents
| Source: HF Papers | Original article
Work2Skill captures human GUI demonstrations to enable reusable skill induction for computer-use agents, tackling the challenge of unfamiliar software and workflow knowledge.
A new open‑source framework called **Work2Skill** aims to bridge the gap between human‑performed GUI work and reusable AI‑agent skills. The system records a user’s screen, mouse clicks, keyboard strokes and accessibility‑tree data in a synchronized “WorkRecorder” log, then lets an agent inspect the multimodal evidence, draft a skill definition and iteratively refine it. The authors demonstrate the approach on the MyPCBench benchmark, where Work2Skill‑generated skills achieve full‑credit success, and they release a 350‑task demonstration dataset on Hugging Face (WorkDemo‑350) together with the code on GitHub.
Why it matters is twofold. First, computer‑use agents have become adept at routine interface actions, yet they still stumble when confronted with unfamiliar software that requires domain‑specific procedural knowledge. By capturing human demonstrations directly in the GUI space, Work2Skill provides the contextual grounding that pure code‑oriented training lacks. Second, the framework’s agent‑directed authoring loop—where the AI can request more evidence, revise its draft, and re‑read the result—offers a scalable path to building libraries of self‑contained, reusable skills rather than training monolithic models for each new application.
Looking ahead, the community will watch how well the released dataset supports broader skill induction across diverse operating systems and enterprise tools, and whether the approach can be integrated into existing agent platforms such as Meta’s Muse Wonder suite. Further research will likely explore automated validation of induced skills, extensions to multi‑step workflows, and the impact of such reusable skill libraries on the “verification tax” that currently hampers AI‑augmented work.
Sources
Back to AIPULSEN