How Firefox's AI Harness Fixed 500 Security Bugs in a Month

Jun 22, 2026 · Lenny's Podcast
🎧 PodShort 48 min squeezed to 2 AI SprinklerAS Sales Tech
Episode artwork
Brian Grinstead
Distinguished Engineer at Mozilla Firefox
Lenny's Podcast
48 min squeezed to 2
Full episode from Lenny's Podcast
Quotable Moments

I think people really underappreciate the relentless tedium that an agent will go through. And the ability to take an agent and give it a very constrained problem and surface area and say, exhaust every attempt at this is really powerful.

Claire VoProduct Leader and AI Obsessive
10:39

What was gated before was just on discovery. It's really hard to find these bugs. But our goal is not to have a bunch of bugs that are hard to find. Our goal is to have zero bugs.

Brian GrinsteadDistinguished Engineer at Mozilla Firefox
44:13

The thing that makes this different is that we have this... This is exactly the sort of shape of a bug report that we send on to our engineering team.

Brian GrinsteadDistinguished Engineer at Mozilla Firefox
21:15
Key Insights
  • Firefox's success in fixing a deluge of security bugs is largely due to its custom 'harness' and integration into existing pipelines, not just model improvements.
  • AI agents excel at repetitive, tedious tasks (like trying hundreds of solutions), where human cognitive energy would decline, making them ideal for exhaustive search.
  • Guardrails and verifier sub-agents are critical to constrain AI agents, preventing them from achieving goals in unintended ways or introducing new issues.
  • The multi-agent verification process virtually eliminates false positives, addressing the 'slop problem' of unactionable bug reports.
  • The AI agent independently reasons how to trigger a bug, simulating attacks and generating exploit test cases (e.g., HTML) that are fed into existing fuzzing infrastructure.
  • AI agents can perform 'code archaeology' by executing obscure 'git' commands and semantically tracing when a bug was introduced, which is difficult for humans.
  • The AI agent generates a functional test case (e.g., an HTML page) that precisely replicates the bug, proving its existence and enabling engineers to confirm the fix.
  • Firefox uses a 'simple LLM judge' to prioritize files for AI analysis based on likelihood of memory safety issues and ease of access from a webpage.
Metrics Mentioned
  • 500 security bugs (Firefox fixed almost 500 security bugs in one month using AI.)
  • Tens of thousands of source code files (The size of the Firefox codebase.)
  • Tens of millions of lines of code (The size of the Firefox codebase.)
  • 14 times (The number of attempts an AI agent made to find a specific bug.)
  • 15-year-old bug (The age of a specific XSLT bug found by the AI agent.)
  • 30% faster (Metaview customers close hiring roles 30% faster.)
  • 100 engineers (The number of engineers mobilized during an incident response event at Mozilla.)
  • 60 new bugs (The number of bugs found during an incident response event at Mozilla.)

RevBots.ai View:

  • AI Sprinkler teams should study Firefox's verifier sub-agent approach to prevent AI slop
  • The 14-attempt bug hunt shows AI's advantage in relentless tedium over human engineers
  • ARM-stage companies could repurpose this for sales tech: imagine AI testing every possible demo flow
  • Their git archaeology proves AI can surface forgotten technical debt in CRM implementations
🎧Full Episode:Lenny's Podcast →