Why We Don't Fully Trust Our Own Benchmark

XploitScan's public benchmark currently sits at 97.0% recall across a corpus of 615 labeled code fixtures that we keep adding to (the benchmark page has the live count). On the same corpus, Semgrep's community rulesets catch about a fifth of the vulnerabilities and Bearer catches about two in five.

Those are real numbers. Checked daily and rerun when rules or fixtures change (at least weekly); updates go live with the next deploy. The per-rule results and the competitors' tool versions are published on the benchmark page. We're proud of them.

And you should be a little suspicious of them. We are.

The problem with acing your own exam

A detection benchmark is only as honest as its corpus. Ours is hand-curated: when we write a rule, we write fixtures — a vulnerable example the rule should catch, and a safe counter-example it must not flag. That's genuinely good engineering. Every false-positive fix ships with a clean fixture that locks the fix in, which is how we keep rules from firing where a fixture's label forbids it while adding rules.

But it has an obvious failure mode: the people who wrote the rules wrote the test. A fixture written by the rule author tends to look like the pattern the rule already matches. Score 97.0% on that and you've proven the rule catches the thing it was built to catch — not that it catches the same vulnerability when a stranger writes it three ways you didn't think of.

This is the single most fair criticism of any security scanner's self-reported numbers, and we'd rather raise it ourselves than have a skeptical buyer raise it for us.

Two things that keep the number honest

It's a floor, not a trophy. A perfect score on a small corpus is not a claim that we catch everything in production code. It's the line below which detection quality is not allowed to regress without us noticing. Checked daily and rerun when rules or fixtures change (at least weekly); updates go live with the next deploy. On pull requests that change rules or fixtures, a per-rule F1 drop of more than two points fails CI. The corpus keeps growing, so the test gets harder over time, not easier. You can watch the trend on the benchmark page.

The numbers exclude our AI filter. XploitScan has an optional AI false-positive reviewer, but the benchmark scores the deterministic scanner with nothing on top — so the precision number is deterministic: no API key and no model-version variance. Turning the filter on can only remove borderline findings, never add them. The published precision is the worst case, not the best.

The real fix: code we've never seen

The only way to answer “but does it work on code your rule authors didn't write?” is to test against exactly that. So we're building a held-out corpus from public, intentionally-vulnerable projects that no one on our side authored: OWASP NodeGoat, Juice Shop, the Damn Vulnerable NodeJS Application, and a prototype-pollution example.

These are real files with real, documented vulnerabilities — SQL injection through a raw Sequelize query, command injection through exec, XXE through libxmljs with external entities enabled, insecure deserialization, prototype pollution, hardcoded crypto keys — written by other people, years before our rules existed. We strip the hint comments these teaching projects include (so the scanner can't cheat off a // vulnerable marker) and score against the bare code.

We expect our recall to be lower on that set than on our own fixtures. That's the point. A held-out number that's a little worse and a lot more believable beats a self-graded number that's perfect and easy to dismiss. When it lands, it'll be on the same public benchmark page, scored the same way, with the sources cited.

Update: the held-out set launched on June 10 at 6 of 10. Over the next day we changed rules to catch the misses we had studied, and we kept doing that as cases were added, so today's held-out figure includes cases we tuned against and is no longer a fully blind number.

Why we're telling you this

Most security tools show you a precision number and dare you to disprove it. We'd rather publish the per-rule results, the competitors' tool versions, and the most honest critique of our own methodology, and let you check. The whole pitch of a scanner is that you can trust what it tells you. That trust has to start with how we talk about our own results.

Want to see where the numbers stand right now? The benchmark page is live. Checked daily and rerun when rules or fixtures change (at least weekly); updates go live with the next deploy. Or just point the scanner at your own code — npx xploitscan@latest scan ., no signup, runs locally.