The failure mode of AI-assisted security work is not missing bugs. It is producing confident, well-written reports for bugs that were never there.
Point a capable model at a codebase and ask it for vulnerabilities and it will give you vulnerabilities. It will give you a plausible sink, a plausible source, a severity rating, and a paragraph of reasoning that reads exactly like the real thing. Some of them will be real. The rest are the expensive part.
This is the actual problem with autonomous offensive security, and it is not solved by a better model. It is solved by refusing to let anything out of the pipeline that has not been executed.
The gate
Our rule is blunt: a finding is not a finding until someone — a triager, a maintainer, anyone who is not us — can re-run the proof and watch it work on a clean build. Not a description of an exploit. The exploit.
- It reproduces from a clean checkout, with the setup steps written down.
- It comes with a negative control: the same input against the patched or guarded path, showing it does not fire. Without that, you have not proven which code you reached.
- The evidence is independent of the thing being tested. A proof-of-concept that validates itself is not evidence, it is a mirror.
- The severity is what the impact supports, not what the report would be worth if the severity were higher.
The third point is the one that catches people. If your test harness mocks the backend, your timing measurements are fiction. If your scanner reports its own output as confirmation, you have built a loop that agrees with itself. We have shipped both mistakes and had to retract them, which is how they became rules.
What this costs
It is slow. A candidate that would take thirty seconds to write up takes a day to actually stand up: build the target, reach the code path, prove the guard was the thing that failed. Most candidates die there, and that is the point — they were going to die at the vendor instead, with our name on them.
The cost of a false positive is not the time you spent on it. It is the attention you will not get next time.
Vendor security teams triage on reputation. A researcher who has sent ten reproducible findings gets read carefully. A researcher who has sent two good ones and eight confident-sounding guesses gets read last, forever. For an automated pipeline that can generate candidates faster than any human can check them, this is an existential constraint, not a nice-to-have.
Where the agent actually helps
None of this makes the automation useless — it changes what the automation is for. The agent is very good at the part humans are bad at: reading everything. Every caller of a fixed function, every sibling repository that vendored the same helper, every place a guard string appears in the history and then quietly stops appearing.
It is worst at the part humans are good at: knowing when a story is too neat. So we let it generate widely and we gate hard at the exit. The pipeline's output is not reports. It is a queue of things worth trying to break, ordered by how likely the break is to be real.
If we cannot prove it, we do not ship it. That sentence is the whole method.