I Scanned 42 Public SaaS Startup Repos. 83% Had Hardcoded-Secret Patterns.

I cloned 45 public open-source repos, mostly from SaaS companies — PostHog, Supabase, Cal.com, Plausible, n8n, Strapi, Medusa, Twenty, and 37 more — pointed our scanner at them, and let it run.

42 of the 45 cloned successfully. Across them I scanned 218,328 source files. After running our most-precise rule and a Claude-based false-positive filter, 35 of the 42 repos (83%) still surfaced matches for our hardcoded-secret rule (VC001), which looks for AWS access keys, Stripe keys, private keys, credentialed database URLs and generic key/token/password assignments.

That number is more interesting than it sounds. A match is not a confirmed key: we never checked whether any of them is a live credential. What's in the noise — and why even a tuned scanner can't fully separate signal from it — tells you more about the state of code-security tooling than the headline does.

Why I ran this

We've spent four rounds tightening false-positive rates on our internal benchmark. On labeled fixtures the scanner now produces 4 findings out of 181 originally flagged (an earlier internal run; current figure: 86.3% precision, Nov 3, 2026 hand review updated Nov 4, on /benchmark), the topic of an earlier post comparing us to Semgrep on a public corpus. But labeled fixtures are tiny, hand-curated samples. Real-world monorepos have thousands of test files, generated code, fixture data, and docs. I wanted to see what the same scanner does in the wild.

The headline finding: even with our most-precise rule and an AI false-positive filter, real codebases still produce a stream of ambiguous matches. That stream is the work item every security team spends most of their time on. It's the actual product problem.

Methodology, in one block

Before any numbers, here's exactly what I did. The full script — corpus-scan.js plus the repo list and README — is published as a public gist; you can re-run it on your own corpus.

• Corpus. 45 hand-picked public open-source repos, mostly from SaaS companies. Selection criteria: a substantial product codebase, an open-source license, and a well-known project. Categories: auth, analytics, CMS, e-commerce, messaging, low-code, observability, etc.
• Scanner. The same regex + entropy engine that ships in our CLI. I cloned each repo shallow, walked source files (skipping node_modules, build outputs, lockfiles, and any file over 1MB), and ran them through the rule engine.
• Rule subset. Six hand-picked, high-precision rules — VC001 (hardcoded secrets, including AWS access keys), VC005 (unprotected Stripe webhook), VC094 (command injection via child_process.exec template literal), VC152 (webhook missing signature verification), VC153 (CORS reflecting origin without allowlist), VC158 (API route serving user-scoped data without an ownership check). Entropy-based secret detection was disabled — too noisy on real codebases without further filtering.
• AI filter. Up to 50 findings per repo ran through the same Claude-based false-positive pass our paying customers see, which classifies each match as real or noise based on the surrounding code; in repos with more than 50, the remainder was counted without AI review.
• Anonymization. No company name, file path, or line number from the corpus appears anywhere in the published output. Stats are aggregated; per-repo identity stays private.

The numbers

Three runs, progressively more filtered:

Full ruleset (206 rules + entropy, May 2026):
  raw findings:        ~12,700  (across 3 sample repos)
  by severity:         2,334 critical / 6,512 high / 3,036 medium / 818 low
  median per repo:     4,034
  signal-to-noise:     dominated by FPs

Trusted subset (6 hand-picked rules, no entropy):
  raw findings:        2,069  (across 42 repos)
  rules that fired:    1 (VC001 — hardcoded secrets)

Trusted + Claude FP filter:
  surviving findings:  1,355
  AI removal rate:     34.5%  (714 classified as FP)
  repos with ≥1 hit:   35 of 42  (83%)
  median per repo:     8.5
  4 repos:             >100 hits each

The baseline is what the full scanner produces on real code, untrimmed. On a 3-repo sample, the full ruleset produced ~4,000 findings per repo. Most are false positives. This is the noise problem every scanner inherits when it leaves the lab.

On the full corpus the six trusted rules produced 2,069 raw findings across 42 repos. Adding the AI filter on top removed 34.5% of those. We end with 1,355 findings across 42 repos — all from one rule (VC001, our hardcoded-secret rule), all marked critical, distributed unevenly across the corpus:

Repos by surviving-findings count

  0 findings        ████████  7 repos  (17%)
  1–5 findings      █████████  8 repos  (19%)
  6–20 findings     ███████████████████  15 repos  (36%)
  21–100 findings   █████████  8 repos  (19%)
  100+ findings     █████  4 repos  (10%)

Seven repos came back clean. Eight had a handful of hits. The center of the distribution is the 6–20 bucket — 36% of repos. Four repos blew past 100.

Why so many “keys” aren't keys

Among its patterns, VC001 matches AWS access-key IDs (AKIA/ASIA-style prefixes followed by 16 characters, as in AKIA[A-Z0-9]{16}). The pattern is narrow, but on our labeled fixtures it still produces some false positives: 82% of clean fixtures come back with none. On real public SaaS repos, it lights up everywhere. The reasons are mundane:

// from a real test fixture (anonymized):
const TEST_CREDENTIALS = {
  accessKeyId:     "AKIAIOSFODNN7EXAMPLE",  // public AWS docs example
  secretAccessKey: "wJalrXUtnFEMI/K7MDENG/bPxRfiCYEXAMPLEKEY",
};

// from a regression test:
expect(maskKey("AKIAIOSFODNN7EXAMPLE")).toBe("AKIA****EXAMPLE");

// from a tutorial markdown rendered into HTML:
<code>aws_access_key_id = AKIA********EXAMPLE</code>

AKIAIOSFODNN7EXAMPLE is the canonical placeholder from the AWS documentation. It appears in test fixtures, mocks, examples, and tutorials across most cloud-aware codebases. AKIA****EXAMPLE shows up in masking tests. AKIA0123456789ABCDEF in API documentation. These aren't accidental commits — they're intentional, public, used as negative-control data in the codebase's own tests.

Of the 2,069 raw VC001 hits across 42 repos, the AI filter classified 714 of them as obvious FPs (test fixtures, public examples, masking-output assertions). It left 1,355 in place, some of them only because they sat past the 50-per-repo review cap — but “left in place” doesn't mean “real key.” For the ones it reviewed, it means the model couldn't cleanly tell from the surrounding code whether the match was a fixture or production data. We did not check whether any match is a live credential (the filter judges context, not validity), so we make no claim that any specific repository exposes a working key.

That's the actual problem. The signal-to-noise ratio of regex security scanning on real codebases is bad even when the regex is precise. Pattern matching alone can't answer “is this a real secret” without context — what file is it in, is the variable used in production code paths, does anything dereference it for a network call. Static analysis can sometimes get there, but a pattern match on its own can't.

What changes when AI is part of the pipeline

We've been shipping a Claude-based false-positive filter for paying customers since April. The 34.5% removal rate above is what it does, mid-flight, on real code. It looks at the matched string, the surrounding ten lines, the file name and path, and decides whether the finding is “real” or “FP.” It catches the obvious patterns: test fixture, masked example, public placeholder, type-check rather than secret comparison, dev-controlled constant rather than user input.

It's not magic. It missed enough that 1,355 findings still need a human eye. But the survivors are dramatically more triageable than 2,069 raw matches — and dramatically more triageable than the ~4,000 per repo the full ruleset produced on the 3-repo sample.

The lesson, to me, isn't “every scanner needs AI.” It's that the scanner's job has changed. A good scanner now is a fast pattern matcher paired with a slow context understander. The pattern matcher casts a wide net cheaply. The context understander triages. Without that second stage, the user has to be the second stage, and most users — particularly the solo SaaS founders shipping AI-generated code — don't have the time or training for it.

That's the same argument we made when we looked at why traditional SAST tools fail on AI-generated code. The corpus data here is the empirical version of it.

Caveats and limitations

• Sample size. 42 repos is small. The distribution is suggestive, not definitive. We'd need 500+ to make claims about the broader public SaaS landscape with confidence; this is research, not a survey.
• Selection bias. The corpus is hand-picked OSS projects, mostly from well-known SaaS companies. They're likely better-instrumented for security than the median private startup. The real-world findings rate on private codebases is almost certainly higher.
• One rule fired. Of the six trusted rules, only VC001 produced hits. The other five (Stripe webhook unprotected, command injection, generic webhook signature, reflected CORS, IDOR) target very specific patterns; their absence on this corpus is expected and matches the precision-first design.
• Per-repo review limits. The AI filter reviewed at most 50 findings per repo. In repos with more raw matches than that, the tail wasn't AI-reviewed and is included in the published count as-is — so the real-vs-noise split on those is approximate.
• Anonymization isn't perfect. Categories with one repo (e.g. crm, scheduling, automation) are guessable from the corpus list. We rolled them up where it mattered; readers paying close attention can deduce a few.

If you want to run your own

The corpus-scan script lives as a public gist — the runner, the curated repo list, and a README. It takes a list of repo URLs and produces an anonymized JSON exactly like the one this post is written from. If you maintain a SaaS company and want to know what your codebase looks like to a scanner like ours, you can run our scanner on your private repo too — npx xploitscan@latest scan . uses the same engine without the corpus harness.

And if you'd rather skip the scanner and just want the artifact procurement teams ask for — a public Trust Page with your security stance, security contact, subprocessors, and self-attested compliance — we just shipped a free builder for that. Two minutes, no signup.

Frequently asked

npx xploitscan@latest scan .