GitHub Introduces whatbroke: A Tool to Compare AI Agent Behavior Across Different Runs
agents
| Source: Mastodon | Original article
New tool helps compare AI agent behavior between runs. It identifies changes in tool calls, arguments, and outputs after model or prompt edits.
A new tool called whatbroke has been introduced on GitHub, allowing users to compare the behavior of an AI agent between two runs. This command-line interface (CLI) tool diffs the behavior, highlighting changes in tool calls, arguments, costs, and outputs when models are swapped or prompts are edited.
Whatbroke is significant because it provides a straightforward way to identify and understand changes in AI agent behavior, which can be crucial for debugging and ensuring consistency. By running `whatbroke diff`, users can pinpoint dropped tool calls, argument drift, and changes in cost and latency, and even set up the tool to fail builds when changes are detected.
As developers continue to work with AI models, a tool like whatbroke can help streamline the process and prevent errors. Users can integrate whatbroke into their continuous integration (CI) pipelines, using the `--fail-on changed` option to automatically fail builds when changes are detected. This can help maintain behavior contracts and ensure that AI agents behave as expected.
Sources
Back to AIPULSEN