A tool that reports nothing and a tool that is silently broken produce identical output. The only way to tell them apart is to plant something it must find.
Every automated security pipeline eventually produces a clean run, and a clean run is the least trustworthy output a tool can give you. It is indistinguishable from a tool that was pointed at the wrong directory, given a filter that excluded everything, or handed an argument name it silently ignored.
So before any scan is allowed to return "nothing found", it has to find something we put there on purpose.
The positive control
Plant a detectable artefact in the exact place the real thing would live, run the pipeline unchanged, and confirm it fires. Then remove it and run again. Only the second run's silence means anything.
# 1. plant, in the same tree and format the real thing would take
$ printf 'AKIA%s\n' 'IOSFODNN7EXAMPLE' > fixture/creds.env
# 2. run the pipeline exactly as it runs in production — no extra flags
$ ./scan.sh ./fixture
[!] fixture/creds.env:1 aws-access-key-id
# 3. only now does an empty result on the real target mean anything
$ rm fixture/creds.env && ./scan.sh ./target
(no findings)It looks like busywork until the first time it catches something. In our own tooling it has caught: a scanner that cannot read the repository format we were handing it and reported success anyway, an argument whose name differed in case from what the engine read, and an option that was accepted, ignored, and never warned about.
- Each of those produced a clean run.
- Each clean run was wrong.
- None of them logged an error.
The same idea, one level up
This generalises past scanners. Any claim that something is absent needs a check that would have detected it if present. "No hardcoded credentials" needs the grep that proves the grep works. "That code path is unreachable" needs the instrumentation that fires when it is reached.
It is the same discipline as a negative control in a proof-of-concept, pointed the other way: there you prove the vulnerable path is what fired, here you prove the detector would have fired at all. Both exist because a confident report and a broken tool look the same from the outside.
An absence claim is a measurement. Measurements need calibration.