We Ran Semgrep Against Our Benchmark. It Missed Half the Bugs.

Every time we show someone XploitScan, we get the same question: “Isn't this just Semgrep?”

Fair question. Semgrep has been around for years, it's genuinely good, and most of its community rulesets are free. So we ran it against our own detection benchmark to see how it actually compares. The benchmark is public, the runner is public, and anyone can reproduce the numbers. Here's what we found.

The numbers

On 26 labeled fixtures covering SQL injection, XSS, command injection, eval, unvalidated redirects, reflected CORS, missing pagination, and hardcoded secrets (AWS, Anthropic, GitHub PAT, Slack, Supabase service role) — XploitScan detected all of them. Semgrep detected 5 of 10 vulnerability classes.

All the methodology, per-rule scores, and reproducibility details live on the /benchmark page. Short version: a rule counts as “covered” if any finding from that scanner lands within ±10 lines of the expected vulnerability. We don't require rule-ID equivalence — the question is can the tool detect the class of bug, not is the rule-name the same. That methodology actually favors Semgrep, because one Semgrep rule firing on a fixture counts as a hit even if it's a vaguely-related finding.

What Semgrep catches

We don't want to undersell Semgrep. Its community rules detected:

• React XSS via dangerouslySetInnerHTML
• eval() on request input
• Express res.redirect(req.query.x) open-redirect
• child_process.exec with user input
• Reflected-origin CORS with credentials

These are the classic OWASP patterns Semgrep was designed around. If your code uses these specific shapes, Semgrep will flag them. That's not nothing.

What Semgrep misses

The interesting part is what didn't fire — and why. Five vulnerability classes where Semgrep returned zero findings:

1. SQL injection via template literals

This is the one that matters, because it's a pattern AI coding tools produce. AI coding tools often produce something like this for “fetch a user by email”:

// The kind of query AI assistants often produce for a user lookup:
export async function getUserByEmail(db, email) {
  const result = await db.query(
    `SELECT * FROM users WHERE email = '${email}'`
  );
  return result.rows[0];
}

This is textbook SQL injection. An attacker passes ' OR 1=1-- as the email and dumps your whole users table. Every security-aware developer has seen this pattern a thousand times.

Semgrep's community JavaScript rules miss it. All five SQL-injection variants in our benchmark — pg, mysql2, Prisma $queryRawUnsafe, Drizzle sql.raw, knex .raw — fire zero Semgrep findings. Each of them is an actual SQL injection. Each is a pattern we see in AI-generated code.

Semgrep's own rule messages point you at the fix, the parameterized form:

// The parameterized form Semgrep's rule messages recommend:
const result = await db.query(
  "SELECT * FROM users WHERE email = ?",
  [email]
);

So why the miss? Those rules model specific shapes (e.g. a pg client created via require('pg') feeding .query()), and none of the community rulesets we ran fired on our template-literal fixtures.

2. Service-specific secret detection

Semgrep's community secret rules are regex detectors for well-known formats. Its AWS access-key (AKIA...) and Slack-token detectors ship in p/secrets, which wasn't among the rulesets we benchmarked. We found no community detector for Anthropic API keys (sk-ant-api03-...) or Supabase service-role JWTs.

Keys without a prefix-specific rule fall through to generic API-key/secret patterns, and those trade recall for false positives. Specific prefix rules don't have that tradeoff.

3. Configuration and availability bugs

None of the community rulesets we benchmarked check for these two patterns, which matter for AI code:

• Missing Stripe webhook signature verification. The vulnerability isn't that any specific function is called unsafely — it's that stripe.webhooks.constructEvent isn't called at all. Semgrep can express absence (its own Express CSRF-middleware rule uses pattern-not-inside), but no rule in the community packs we ran checks for a missing constructEvent call.
• Missing pagination on a list endpoint. Semgrep sees prisma.item.findMany({ where... }) with no take argument. A rule could flag that, but none of the community packs we ran does.

XploitScan rules for both patterns look at the shape of the handler — route declarations, function bodies, expected protective calls — not just individual lines. It's narrower than a taint analysis, but broader than regex.

The honest caveats

A few things this comparison deliberately does not claim:

• Semgrep's Pro rules would likely score higher. Those rules and cross-file analysis are free on Semgrep's platform for up to 10 contributors and 10 repositories. We tested the community rulesets because they run without an account. XploitScan was benchmarked with its full rule set as it stood in April (what the paid tiers run); the free tier runs 30 rules.
• A small corpus is not the full story. A thousand-fixture corpus would give both tools more room to shine and expose more gaps on both sides. The corpus was already at 41 fixtures when this post went up and has kept growing since — see the benchmark page for the current numbers.
• Semgrep is extensible. If you write your own rules tuned for AI-generated patterns, it can match XploitScan on any specific vulnerability. The benchmark measures out-of-the-box behavior with the community rulesets.
• This benchmark is ours. We wrote the fixtures, we wrote the runner, we wrote the scoring methodology. A different corpus would produce different numbers. If you think our corpus is biased, the fixtures are open on GitHub — open a PR with a fixture you think we miss.

When to use which

Semgrep and XploitScan are not really competitors — they're aimed at different audiences for different use cases. Rough heuristic:

• Semgrep if you have a security engineer writing and maintaining custom rules for your specific codebase, you need cross-file dataflow analysis (Pro), or you're on a language XploitScan doesn't deeply support yet.
• XploitScan if you're shipping a product built with Cursor, Bolt, Lovable, or Replit and you want out-of-the-box detection for the specific bugs AI coding tools produce, without rule-writing overhead.

They stack. Run both, dedupe findings, ship a safer app. If you do, you'll notice XploitScan catches the bugs Semgrep misses, and Semgrep occasionally catches something XploitScan missed (we fold those back into our rules when they do).

Reproduce the numbers

Everything in this post is in a public repo. To reproduce:

1. Clone github.com/bgage72590/xploitscan-benchmark and run npm install
2. Run node scripts/benchmark.js for XploitScan's numbers
3. Install Semgrep (pip install semgrep==1.86.0, the version behind the published numbers) and run node scripts/semgrep-benchmark.js for the comparison
4. Both scripts write JSON to the repo root and the /benchmark page renders whatever the latest runs produced

Checked daily and rerun when rules or fixtures change (at least weekly); updates go live with the next deploy. The benchmark page shows the latest published JSON, so it can lag the most recent run.

npx xploitscan@latest scan .