Vibe Security Index — Methodology

What the index measures

The Vibe Security Index answers one narrow question: across public repositories generated by a given AI app builder, how often does the application code contain a finding serious enough to matter? That is it. It is not a review of the builders' products, their infrastructure, their hosted runtime, or the security of their own systems. It is a measurement of shipped code that happens to carry a builder's fingerprint.

The reason to publish it is that the alternative is anecdote. “AI-generated apps are insecure” is repeated constantly and almost never with a number attached, and the people repeating it usually sell something. We sell something too — which is exactly why the methodology is published in full, why the limitations sit next to the chart rather than under it, and why we publish the size of the data we excluded alongside the data we kept.

Rounds are labeled by month (for example 2026-07). Each round is a fresh sample and a fresh scan; rounds are not cumulative and a builder's number moving between rounds is at least as likely to be sampling noise as real change.

Where the sample comes from

Finding the repositories

Each builder leaves a scaffold marker in the projects it exports, though not all of them put it in the same place. Lovable and Base44 write theirs into package.json (a build-time dependency and an SDK entry respectively); v0 merges a generated config into the project's Next.js config file; Bolt records its template id in .bolt/config.json. Those markers are public and unambiguous, and they are the only mechanism we use to attribute a repository to a builder. The exact marker in use for each builder is printed on the index page next to that builder's row, so the attribution step is reproducible by anyone with a GitHub account.

We enumerate public repositories containing the marker through GitHub code search. Code search caps how many results any one query returns, so the universe size shown on the index page is a floor on the true population, not a census — the real number of public projects from a given builder is at least that large and probably larger.

Choosing what to scan

From the matched set we take a sample, clone each repository at depth 1, and scan it. Nothing in a cloned repository is ever executed: no dependency install, no build step, no test run, no script. The scanner reads files; that is the entire interaction. This matters both for our safety and for the integrity of the result — a build step could fetch code that was never in the repository we claim to have measured.

The sample sizes, the number of repositories excluded, and the number actually scanned are all printed on the index page. If the scanned count is small relative to the universe, the confidence interval around that builder's rate is wide, and the table should be read accordingly.

Eligibility and exclusions

A sampled repository is scored only if it is actually a project. The live eligibility list for the current round is rendered on the index page itself — it ships in the same data file as the numbers, so it cannot drift away from what was run. In broad terms:

• The marker has to appear where the builder actually writes it — a dependency manifest or a generated project file. A marker mentioned in a README, an issue, or a transitive lockfile entry is not attribution.
• The repository needs at least one file the scanner can analyze after ignore rules, binaries, and lockfiles are removed.
• It needs at least one non-vendored application source file. A repository that is nothing but generated scaffold tells you about the scaffold, not about anyone's app.
• Starter templates, example projects, and demo repositories published by the builder platform itself are excluded. They are marketing artifacts, not shipped software, and they would be counted hundreds of times over through forks.

Exclusions are counted and published per builder. A methodology that silently drops repositories is a methodology that can be tuned until the chart looks right.

Application code vs. vendored scaffold

This is the single most important decision in the whole study, and the one most likely to be gotten wrong in a way that produces an exciting, useless headline.

Modern app builders scaffold a project by copying in a large set of UI component primitives from public component libraries. Those files are not written by the builder and not written by the developer — they are the same files, frequently byte-identical, in thousands of unrelated projects. A repository can easily ship fifty or sixty of them before a single line of application logic exists.

If you scan that scaffold and charge the findings to the builder, you get a dramatic number that measures a component library — the same number, for every builder that vendors the same library, no matter what their code generator does. And a meaningful share of those findings are our false positives: a chart primitive that injects a <style> block assembled from a module-level theme constant is CSS-variable theming, not an injection sink. A rule that flags it is a bug in our rule, and pointing at it as evidence about somebody's security posture would be indefensible.

So every finding is assigned to one of two buckets by path, before any headline number is computed:

• Application code — everything that is not in a declared vendored path. All headline figures (critical rate, high rate, median findings, grade distribution, top recurring rules) come from this bucket and only this bucket.
• Vendored scaffold — component-library directories, build output, minified bundles, and vendor directories. The exact glob list for the round is printed on the index page and shipped in the data file.

The split is mechanical and path-based. It does not depend on whether an individual finding is a true positive, and it is applied identically to every builder. Vendored findings are still counted and still published — in their own clearly separated section — because deleting them entirely would just be a different way of hiding the data. They never contribute to a grade, a rate, or a ranking.

The cost of this choice is real: it makes the headline numbers smaller and less shareable. That is the point.

Attribution confidence

Every builder row carries an attribution confidence label, because “this repo was made with X” is a much weaker claim than it sounds.

• Scaffold — the project skeleton was generated by this builder, established by a marker the builder writes into the project. This is the level used today for every builder in the index. It says nothing about who wrote any particular line afterwards.
• File — individual files are attributable to the generator by a marker inside the file itself. Stronger, and rare; most builders do not emit anything of the kind.

At scaffold confidence, the honest reading of a row is: projects that started in this builder and were published publicly have this security profile — not this builder writes insecure code. Those are different sentences and the difference is the whole reason the label exists.

Scanning and scoring

Repositories are scanned with the deterministic XploitScan rule set — the same engine, same rules, and same grading function behind a scan you run yourself and behind the CLI. The optional AI false-positive filter is not used: it needs an API key, it introduces model-version variance, and a study nobody else can reproduce is not worth publishing. The rule count for the round is printed in the page header.

Grades use the standard grading function, which caps a project at D once it has a single critical finding. That is why the critical rate is the headline metric rather than a mean finding count: for the person shipping the app, the difference between zero criticals and one critical is the difference between “fix it this quarter” and “fix it before you tell anyone the URL.”

Detection quality is itself measured and published. The scanner's precision and recall against a labeled corpus, plus a held-out corpus of code we did not write and a head-to-head against other scanners, are on the benchmark page, including the vulnerability classes it currently misses. Every rate in the index inherits those error bars.

Limitations — read this before quoting a number

We publish this list with the same prominence as the results. Each round's data file carries its own copy so the caveats travel with the numbers.

Public repositories are not a random sample

This is the biggest one. People publish side projects, demos, tutorials, and portfolio pieces. The apps that hold real customer data and real payment flows are disproportionately private, and nothing about them is in this sample. A public repo is also more likely to have been scrubbed of secrets before it was made public — and more likely to have been abandoned mid-build. Both effects exist and they push in opposite directions, which means we cannot even tell you the sign of the bias, only that it is there.

A marker identifies a scaffold, not an author

The marker proves the project was generated by that builder. It does not prove the builder wrote the vulnerable line. Plenty of these projects were exported once and then developed for months by hand, or in Cursor, or in Claude Code, or by a contractor. We attribute findings to the repository, never to a line's author, and the index should be read as “projects that came out of this tool” rather than “code this tool wrote.”

Multi-tool repositories

Related, and worth separating out: a repository can carry more than one builder's fingerprint, and most serious projects pass through several tools. We do not attempt to deconvolve that. A repository matching two markers is counted for each, which double-counts a small number of projects across rows and is one more reason not to read a two-point gap between builders as meaningful.

Static analysis is not exploitation

A finding is a code pattern matching a rule. It is not a proof of exploitability, and nothing in this study is verified at runtime — we never run the code. Some findings are false positives. Some real vulnerabilities are missed entirely, including whole classes our rules do not cover yet. The critical rate is the rate at which our scanner reports criticals, which is a proxy for, not a measurement of, real risk.

Sample size and snapshots

Each repository is scanned once, on its default branch, on the round's date. Issues may have been fixed since; issues may have been introduced since. Sample sizes are printed per builder, and where a sample is small the difference between two builders is usually noise. If two rows are close together, treat them as tied.

We are not a neutral party

We sell a security scanner. A study showing that AI-generated apps have security problems is convenient for us, and you should weigh it accordingly. What we can offer in exchange is that the methodology was written down before the numbers existed, the exclusions are published with their sizes, the scanner's own error rate is public including the cases it fails, and every builder is invited to challenge a number and have the round re-run.

Ethics — aggregate only

Everything published from this study is an aggregate. We do not publish repository names, owners, URLs, stars, or any other identifier, and we never publish a code snippet, a secret value, or a finding tied to a project. Nothing that could function as a target list leaves our side.

The internal per-round data is anonymized at write time: each repository is stored under an opaque identifier with its file count, finding counts, and grade — no name, no owner, no URL, no code. Cloned repositories are never executed, installed, or built.

If a scan surfaces something that looks like a live, exploitable exposure in a specific project, it is handled privately under our disclosure policy and never in a chart.

Corrections and right of reply

If you build one of the platforms in the index and believe a number is wrong — a marker that over-matches, a vendored path we missed, a rule producing false positives on your template — email admin@xploitscan.com. We will re-run the round with the correction applied and publish the result whether it moves in your favor or not. A correction that changes a published figure gets a note on the index page and an entry in the changelog.