Tomosu produced 114 reliability findings on a snapshot of the open-source Supabase repository. Fifty-nine of those corresponded to defect classes and code areas already represented by public GitHub issues — evidence that standing-code analysis can expose reliability risk before the next pull request happens.
What Tomosu is. Tomosu is the Production Reliability Governance layer: it evaluates standing code, weighs findings by blast radius, and applies risk-based policy at the merge gate.
What we tested
What we validated: Do Tomosu's reliability findings correspond to independently documented issues in Supabase's public issue history?
We scanned a fixed snapshot of the supabase/supabase repository, then compared each finding with public issues in the review dataset drawn from the project's GitHub issue tracker. A match required the same defect class in the same code area. That definition matters: we did not count similar user symptoms alone as proof of a match.
Continuously assess the standing codebase, measure the reach and criticality of each finding, and apply the right evidence and policy before a change reaches production.
What the validation produced
The scan returned 114 reliability findings. Fifty-nine corresponded to at least one public issue in the reviewed history. Because findings and issues form a many-to-many relationship, those 59 findings mapped to 48 distinct issues through 101 finding-to-issue links.
"No public match found" is not evidence that a finding is a new defect, nor that it is a false positive. Those candidates need independent engineering adjudication. The unmatched set cannot support a precision or recall claim.
The Tomosu dashboard: PRI 31 → 86
The baseline scan recorded a Production Reliability Index of 31/100 with all 114 recommendations pending. The second captured state shows all 114 recommendations accepted and a PRI of 86/100.
A Supabase TypeScript/React monorepo containing Studio observability, realtime, storage, AI cron generation, common auth utilities, and PostgreSQL metadata tooling. It serves dashboard users and support teams who need reliable project operations, secure metadata access, and diagnosable failures across browser, Next.js, TanStack Router, and pg-meta packages.
| Recommendations by category | Total | Baseline | Governed |
|---|---|---|---|
| Support Agent Readiness | 45 | 45 pending | Done |
| Data Integrity | 34 | 34 pending | Done |
| Exception Handling | 25 | 25 pending | Done |
| Maintainability | 21 | 21 pending | Done |
| Logging | 19 | 19 pending | Done |
| User Facing Diagnostics | 19 | 19 pending | Done |
| Reliability | 18 | 18 pending | Done |
| Total | 114 | 114 pending | 114 accepted |
Critical finding: blast radius changes the engineering decision
Two three-line defects are not equivalent when one sits behind a shared SQL boundary and the other is isolated to a leaf component.
The Supabase analysis measured one SQL formatting/escaping boundary — the pg-format layer — as referenced across 372 files, with 822 raw SQL or escaping call sites in the scanned snapshot. A separate command-menu boundary had one consumer.
The local code may look similar. The governance response should not. A change under a shared query or escaping boundary may warrant stronger tests, domain review, or a temporary block. A leaf change may pass with normal evidence.
The lines of code may look similar. The production risk does not.
Why findings and issue reports do not map one-to-one
A public issue describes an observed symptom. A standing-code finding describes a code-level condition. The mapping between them is often many-to-many.
One swallowed error can surface as a frozen interface, a rejected token, or a blocked sign-in. In the other direction, one broad report such as "failed to fetch permissions" can cover several independent validation, lookup, and error-propagation failures.
This is why the study counts three different objects: findings, issues, and links. Collapsing them into a single "bug count" would make the result easier to market and less useful to engineers.
The harder failures are silent
The public reports that best explain the governance problem are not crashes. They are cases where the system continued to run while presenting or persisting the wrong result.
Cron editor corrupted headers containing commas or parentheses
Saving an edited job could silently rewrite the stored command even when only the schedule changed.
View on GitHub Supabase issue #47015Queue pagination hid messages sharing a timestamp
The interface stopped at 30 rows while additional messages still existed in the queue.
View on GitHubThese are reliability failures because the system violates an implicit guarantee without an obvious alarm. They are expensive in production: operators distrust the interface, support teams investigate inconsistent state, and engineers reconstruct the path from symptom back to code.
A system that keeps running can still be wrong. Production Reliability Governance must reason about violated guarantees, not only exceptions.
The issue record independently documents the same affected code areas and defect classes surfaced by Tomosu.
The most unreliable feature
In the validated slice of Supabase issue history, cron was the most failure-concentrated feature.
Public reports included a cron editor that lost bodies and headers when values contained backslashes in issue #45674, followed by comma and parenthesis handling that silently rewrote stored headers in issue #46829.
Tomosu also found validation and recovery weaknesses in the same feature family at ai-commands/src/sql/cron.ts:8, pages/api/ai/sql/cron-v2.ts:30-34, and data/database-cron-jobs/database-cron-jobs.utils.ts:7-11. This is feature-level corroboration, not a claim that every finding shared the same root cause.
After the second regression, cron should stop being treated as an ordinary feature.
"Most unreliable" describes the concentration observed in this validated issue slice. It is not a platform-wide ranking of every Supabase feature.
What the gate would have done
Tomosu governs the decision, not the code.
It would not tell Supabase how to rewrite the parser. It would turn the scan evidence and repeated production history into a higher bar while the next cron change was still holdable.
The published scan provides a repository PRI of 31/100, not a separate per-finding PRI. The defensible policy action is to raise cron's feature criticality after repeated regressions and require stronger evidence on the next change.
Every one of these was a deliberate decision that nobody got to make, because nothing surfaced the risk while the change was still holdable.
Why this is Production Reliability Governance — not another scanner
A pull-request reviewer evaluates a proposed change. A whole-tree scan evaluates standing code. Neither scope subsumes the other.
Latent reliability conditions can remain in already-merged code until a user hits a path, a dependency changes, or a later pull request changes the surrounding assumptions. This does not prove that a reviewer "missed" the condition. It may never have been inside the reviewed diff.
Production Reliability Governance starts when those signals change a decision. A finding is not a control. A score is not a control. Governance needs an explicit policy at a boundary where the organization can still request evidence, route review, or stop progression.
- 1Detect: identify a reliability condition in standing code or a proposed change.
- 2Contextualize: attach ownership, dependency reach, criticality, and prior production evidence.
- 3Decide: allow, request evidence, route to a reviewer, or block according to policy.
- 4Learn: use post-deployment signals to update future thresholds and decisions.
The Production Reliability Governance loop
The scan is useful because it creates a baseline. The control appears when that baseline follows every change from code through production.
The Production Reliability Index summarizes that posture for decision-making. It is not a bug count. It measures whether the standing codebase and the changes entering it satisfy the organization's production reliability expectations.
The question is not "Did the scanner find something?" It is "What evidence should this change require before it reaches production?"
Govern production reliability — not just a code diff. A reliability signal becomes valuable when it changes the evidence required at merge, and when production outcomes improve the next engineering decision. The full validation ran against the open-source supabase/supabase repository.
Discuss the validation → · Questions: contact@tomosu.ai