Fuzzers vs. AI: Fuzzilli and BigSleep Tested on JavaScript Engines
agents
| Source: Lobsters | Original article
A new analysis pits the JavaScript engine fuzzer Fuzzilli against the AI tool BigSleep to evaluate their effectiveness on JavaScript engines.
Google’s Project Zero has released the first systematic side‑by‑side comparison of a classic fuzzer and an AI‑driven bug‑hunting agent on the same target. Over a six‑month period the team ran Fuzzilli – a generative JavaScript engine fuzzer – and BigSleep – an LLM‑based system that reviews new code and writes proof‑of‑concept test cases – against Chrome’s V8 JavaScript engine. Both tools operate in production, automatically scanning newly added code.
The results show BigSleep identified 45 actionable vulnerabilities, edging out Fuzzilli’s 37. The two approaches overlapped on roughly a quarter of the bugs, indicating each uncovered distinct issues. Project Zero’s blog post stresses that the AI‑powered agent can reason about code patterns and generate targeted test cases, while the fuzzer relies on random input generation.
The finding matters because it challenges the long‑standing dominance of fuzzing in automated security testing. If AI agents can consistently find more flaws than traditional fuzzers, security teams may shift resources toward integrating large‑language‑model tools into their vulnerability‑discovery pipelines. The comparison also provides a concrete benchmark for the emerging class of “agentic” bug finders, a topic that has attracted attention across the industry as AI models become more capable of code analysis.
Looking ahead, Project Zero plans to extend the study to other components of Chrome and to evaluate how AI agents perform at scale. Observers will watch for follow‑up data on false‑positive rates, the effort required to maintain AI models, and whether other vendors adopt similar AI‑driven approaches for their own runtime engines. The experiment marks a pivotal moment in the ongoing debate over how best to secure increasingly complex software with a blend of traditional and AI‑enhanced tools.
Sources
Back to AIPULSEN