What does DETECTED mean in this benchmark?
DETECTED means an AI-generated change has been identified and recorded, but no independent evidence has yet confirmed or refuted the claim. It is a starting state, not a negative judgment.
Research
AI-generated code verification benchmark
AI-generated code claims are signals, not authorizations. The research question is: how many of those claims remain at DETECTED until independent evidence corroborates them?
Methodology
We constructed a synthetic, internal-demo corpus of 410 AI-generated code changes. Each change was recorded with the claim produced by the generation tool, such as "tests pass" or "no security issues found." We then applied a verification rule: AI-generated claims are capped at DETECTED until an independent source corroborates them.
- Claims were generated across a simulated set of repositories and task types.
- Independent sources included CI test results, type checks, security scanners, human reviewer attestations, and runtime telemetry.
- A claim could only be promoted to VERIFIED if at least one corroborating artifact was version-matched and reachable.
Findings
In an internal demo corpus of 410 AI-generated code claims, most remained at DETECTED because the only supporting evidence came from the generator itself. These numbers are synthetic and illustrative, not customer outcomes.
| Verification state | Claims | Share | Evidence basis |
|---|---|---|---|
| DETECTED | 379 | 92% | AI-generated claim only; no independent corroboration |
| PARTIALLY CORROBORATED | 22 | 5% | Passing unit tests or type checks from CI |
| VERIFIED | 9 | 2% | Independent retest, scanner pass, or reviewer attestation |
Synthetic, internal-demo figures. Not representative of customer environments or model performance.
Limitations
- The corpus is synthetic and built for demonstration. It does not use customer code or production data.
- The verification policy applied here is one of many reasonable policies. Other thresholds would change the counts.
- We did not measure code quality, correctness, or security outcomes, only whether claims were independently corroborated.
- The sample size and task distribution are controlled; real-world distributions may differ.
What this does not prove
- It does not prove that AI-generated code is less secure than human-written code.
- It does not prove that AI coding tools are unreliable. It proves that their claims must be corroborated before they authorize a release decision.
- It does not prove an industry-wide verification rate; it is an internal benchmark under a specific policy.
- It does not prove that adding more scanners always helps. Corroboration must be relevant to the claim being made.
Frequently asked questions
Why can AI-generated code claims not authorize alone?
AI-generated outputs are probabilistic. A claim that code is safe, tested, or correct must be corroborated by independent evidence before it can support a consequential release decision.
What counts as independent corroboration?
Independent corroboration comes from a source other than the generator itself: a test result, a type checker, a security scanner, a human reviewer attestation, or runtime telemetry tied to the artifact.
Does this benchmark judge AI coding tools?
No. It measures the verification gap for generated-code claims under a specific evidence policy. It says nothing about the underlying model, tool, or human developer productivity.
Related reading
Doctrine
The public operating doctrine for release-evidence verification.
Platform
How GoSentrix qualifies evidence and evaluates it against policy.
How to prove a vulnerability was fixed
Move from ticket closure to verified remediation evidence.
Validated autonomous remediation
Independent verification for automated remediation claims.
Corroborate every AI-generated claim.
GoSentrix caps AI-generated claims at DETECTED until independent evidence satisfies policy, preserving a clear basis for every release decision.