← Back to docs

Detection Methodology

How XploitScan detects vulnerabilities, how detection quality is measured, and how to read the public benchmark.

Overview

XploitScan combines a fast pattern-matching pre-filter with a taint-aware AST analysis layer. Most rules use the regex layer alone — it's faster and sufficient for things like hardcoded API keys, missing security headers, and configuration mistakes. Rules that depend on data flow — SSRF, SQL injection, prototype pollution, mass assignment, SSTI — confirm matches with a Babel-parsed AST and a local taint tracker.

Detection quality is scored on a public, version-controlled fixture corpus. Checked daily and rerun when rules or fixtures change (at least weekly); updates go live with the next deploy. Numbers are visible at xploitscan.com/benchmark.

The two-layer scanner

Layer 1 — Pattern matching

Every rule starts with a regex. Patterns are tuned per category: file-extension and filename allowlists pre-filter irrelevant files; comment lines and string-literal contexts are stripped; well-known mitigation patterns (e.g. crypto.timingSafeEqual, DOMPurify.sanitize) act as local suppressors.

For pattern-only rules — secrets, missing CSP, weak TLS config, insecure cookie attributes, container misconfigs — this layer is the entire scanner. Fast, deterministic, no parser needed.

Layer 2 — AST + taint analysis

Rules that need to know where data came from — not just that a sink looks dangerous — opt into a Babel parse. The taint tracker recognizes user-controlled sources from Express (req.body, req.query, req.headers), Fastify (request.*), Koa (ctx.request.*), Next.js App Router (await request.json(), request.formData()), Web Fetch API, process.argv, and AWS Lambda event.body. Taint propagates through const/let bindings, destructuring (including renames), assignments, template literals, member access, array-iteration callbacks (.map/.filter/forEach) and for…of loop variables, object/array spread, and taint-preserving calls (a file read from a user-controlled path, or a parser's .parse() of tainted text).

Rules currently AST-confirmed: SSRF, SQL injection, XSS, prototype pollution, mass assignment, SSTI, XXE option-object inspection, open redirect, unsanitized filenames, file downloads without path validation, timing-unsafe secret comparisons, and log injection. The AST pass only runs after the regex pre-filter matches, and the parse result is cached per file so multiple AST rules in a single scan pay the parse cost once.

Language scope. The taint layer is deepest on JavaScript and TypeScript, where a full AST parse drives data-flow analysis. Other languages (Python, Ruby, Java/Kotlin, and more) are covered by the pattern layer today, with data-flow-aware analysis expanding over time. The public benchmark corpus is mostly JavaScript and TypeScript, with a handful of Python, Ruby, Java and config-file (YAML, Terraform, Dockerfile) fixtures; it does not yet report coverage per language.

Suppressing a reviewed finding

When a high-confidence rule fires on a site you've reviewed and accepted, suppress that one finding with an inline comment rather than disabling the whole rule. Add // VC<id>-OK: <reason> on the flagged line, or anywhere in the comment block above it — for example // VC094-OK: arg is a fixed constant, not user input. The wildcard // scanner-OK suppresses any rule at that site. The comment prefix can be //, /* */, #, or -- (SQL).

Every rule honors them, security and quality alike — the marker is applied where findings are produced, not rule by rule. A reason is required by convention so the suppression is self-documenting for the next reviewer; the scanner reads the marker, not the text after the colon.

For broader suppression, a .xploitscanignore file (.gitignore syntax) in your project root excludes whole paths from scanning. A line can also carry a trailing rule-ID list to suppress only those rules on matching paths while still scanning the files for everything else — e.g. scripts/cron/** VC044 or legacy/** scanner (the wildcard suppresses all rules). Reach for the inline comment for a single finding; reach for the ignore file for a whole directory or an ongoing exception.

How quality is measured

The corpus

Every detection claim is grounded in a labeled fixture corpus, stored in the public repo at test-fixtures/. Each fixture is a small directory with one or more code files and an expected.json manifest that lists either the exact findings the scanner should produce or the rules it must not fire.

Fixtures are split into two categories:

  • vulnerable — code that contains a real vulnerability with a known location. Each expected finding declares its rule ID, file, inclusive line range, and minimum severity.
  • clean — correctly written code that exercises the same APIs as a vulnerable fixture (e.g. jwt.verify with a pinned algorithm allowlist as the counter to algorithms: ['none']). Lists rules that must not fire.

Counting rules — TP / FP / FN

  • True positive (TP) — a finding on a vulnerable fixture that matches an expected entry by rule, file, and line (within the declared range).
  • False negative (FN) — an expected entry the scanner did not produce.
  • False positive (FP) — a finding for a rule listed in the fixture's mustNotFire array, or any finding on a clean fixture for a rule that should not fire there.

Other findings on vulnerable fixtures (rules that are neither expected nor explicitly forbidden) are ignored — they may be legitimate adjacent issues, and treating them as FPs would penalize useful findings.

Tracked-but-not-yet-detected entries

Some fixtures document vulnerabilities the current scanner doesn't catch yet — flagged with knownGap: true. They still count as FNs in the public benchmark (so recall stays honest), but the regression test suite skips them so a documented gap isn't treated as a regression while the rule is being improved. The /benchmark page lists a rule in a separate Tracked — in progress section when every one of its fixtures is a known gap; otherwise the gap shows in the rule's FN count.

Numbers are the deterministic floor

The benchmark scores the deterministic scanner — pattern layer plus AST/taint — with nothing on top. The optional AI false-positive filter (in the CLI and GitHub Action it runs only when you supply your own Anthropic API key; web scans and the GitHub App on paid plans run it by default using our key, and signed-in users can turn it off in Settings) reviews findings after the deterministic pass and is not part of the benchmark. So the published precision is a floor: turning the filter on can only remove borderline findings, never add them. We score without it so the numbers stay reproducible by anyone, with no API key and no model-version variance.

Comparing against Semgrep and Bearer

The same corpus is also scanned by Semgrep (community rulesets: p/security-audit, p/owasp-top-ten, p/javascript, p/typescript, p/react) and Bearer (default SAST + security report). Results are normalized into the same TP / FP / FN counts and rendered side-by-side.

Two methodology notes that matter for fairness:

  • Positional coverage, not rule-ID matching. A third-party finding counts as a TP if any of its rules fires on the expected file within ±10 lines of the expected range. We don't expect Semgrep or Bearer to use the same rule names we do; we're measuring whether the scanner can detect the vulnerability class at all.
  • Stricter FP definition for third-party scanners. On clean fixtures, any finding from Semgrep or Bearer counts as an FP — we can't per-rule attribute against mustNotFire the way we do for our own rules. This is the more conservative interpretation and doesn't flatter the comparison.

Reproducibility

Everything in the benchmark is open. The corpus, the runner scripts, the Semgrep config, the Bearer invocation, and the scoring code all live in the public repo:

To reproduce locally:

git clone https://github.com/bgage72590/xploitscan-benchmark
cd xploitscan-benchmark
npm install

# XploitScan benchmark
node scripts/benchmark.js

# Semgrep comparison (requires `pip install semgrep==1.86.0`)
node scripts/semgrep-benchmark.js

# Bearer comparison (requires the `bearer` CLI on PATH)
node scripts/bearer-benchmark.js

Checked daily and rerun when rules or fixtures change (at least weekly); updates go live with the next deploy. The /benchmark page shows the latest published JSON, so it can lag the most recent run.

What the corpus is — and isn't

It is a curated, growing set of labeled examples covering the vulnerability classes the rules target. The corpus holds 615 fixtures (the live count is on the benchmark page), including realistic mini-apps, some multi-file (auth flows, file upload pipelines, payment webhooks, GraphQL APIs, OAuth callbacks, admin dashboards) that exercise rules in integration-level contexts rather than only minimal one-off snippets.

It isn't a replacement for real-world testing on a real codebase. A perfect score on a small corpus is not a claim that the scanner catches everything in production code — it's the floor below which the scoring will not regress without notice. The corpus grows with every release; the score gets harder over time.

Contributing fixtures

Found a vulnerability class XploitScan doesn't catch — or catches incorrectly? Open a PR with a fixture. The format is documented in the test-fixtures README, but the gist is: a directory under vulnerable/ or clean/, your code, and an expected.json with the rule ID + file + line range. The next benchmark run will measure whether the scanner detects it; if not, the fixture becomes a tracked gap and a target for the next rule release.

See also