AI code reliability benchmarks need a denominator

Teams asking how often AI-generated code causes a production incident after QA usually want one number. A number without a denominator, time window and attribution rule is not a benchmark. It is a prompt for the next argument.

That is the practical state of AI code reliability measurement: teams need better comparisons, but a useful local baseline is more attainable than a universal post-QA failure rate. The same discipline applies to security risks. A vulnerability found before merge, a defect that reaches production and an incident linked to a release are different outcomes. Combining them obscures what a team should fix.

Define the unit before comparing rates

“Production incidents per PR” sounds straightforward. It is not until a team answers several questions. Does the denominator include every merged pull request, or only changes that reached production? Is an incident counted once, or are multiple alerts for the same event separate records? How long after deployment can an incident be attributed to a change?

Those choices affect the result. A service with frequent, small deployments will have a different opportunity for attribution than one that ships in large batches. A team that records incidents consistently may appear to have more failures than a team that misses or classifies them differently. Neither number can be compared fairly without context.

A workable internal metric can start small: choose a fixed observation window, define what qualifies as a production incident, and record how the responsible change was identified. Track the number of eligible merged changes alongside the incident count. Keep severity and customer impact visible rather than burying them in one average. The point is not to manufacture precision. It is to make the next month comparable with this one.

Keep security findings in their own lane

AI-generated code security risks also need a clear measurement boundary. A scanner finding is not automatically an exploitable vulnerability, and a clean scan does not prove that a change is secure. Track what was detected, where it was detected, whether a person validated it, and whether it was resolved before merge or later.

Do not roll security findings into a post-QA incident rate. That would mix potential exposure with observed harm. Instead, teams can compare review stages: findings raised before merge, validated issues that escaped, and incidents with a confirmed security cause. A trend is meaningful only if the detection tools and review rules stay sufficiently consistent across the period being compared.

This also changes how teams evaluate AI coding tools. The relevant question is not simply whether generated code passes existing tests. It is whether the team can identify which weaknesses its checks catch, which ones they miss, and whether the escape rate changes as usage grows. That requires a repeatable baseline, not a claim that any single check guarantees safety.

Put verification at the merge boundary

For builders, the immediate response is to make pre-merge verification explicit. Decide which checks must pass for a change to merge, who owns exceptions, and how production evidence feeds the next review. Keep the process proportionate: more checks are not automatically better if they produce noise that reviewers learn to ignore.

A baseline helps separate inherited weakness from new risk. Tomosu AI offers GitHub repository scanning for public and private repositories and calculates a Production Reliability Index across seven sub-indexes. Its free VS Code plugin provides inline reliability findings and fixes without an account; its GitHub App reviews pull requests and provides PRI deltas. Those are measurement and review surfaces, not an industry-wide failure-rate dataset. Teams should treat their output as one input to a defined verification process.

For teams starting from an existing repository, a read-only reliability baseline can help establish what was already present before changing coding practices. The useful comparison comes later: apply the same definitions, review the same classes of change and watch whether validated weaknesses escape into production.

What to watch next month

There is no defensible universal post-QA failure percentage to quote without comparable cohorts and shared measurement rules. For now, the signal worth watching is whether teams publish enough methodology to make their AI code reliability benchmarks interpretable: what counts as a failure, what is included in the denominator, how incidents are attributed, and how security findings are validated.

For an engineering team, the first benchmark does not need to settle the industry debate. It needs to be stable enough to reveal whether a change in tools or review practice is improving outcomes. Count production incidents against a defined set of shipped changes, keep security outcomes separate, and revisit the rules when the workflow changes. That is less dramatic than a headline rate, and more useful at the merge queue.