We put a real Prefect bug back.
Tomosu found it from the logs.

Most tools that claim to diagnose production failures are measured against bugs someone wrote on purpose. We wanted a harder test, so we took a bug that Prefect’s own engineers found and fixed in October 2026, put it back into the code, ran it until it genuinely crashed, and asked Tomosu to explain the crash with no knowledge that a fix existed.

What did we actually test?

A real-bug regression validation is a test in which a genuine, already-fixed defect is reintroduced into a real codebase, triggered for real, and handed to an analysis tool that has not been told what the right answer is. The point is to measure diagnosis rather than recall: a synthetic bug tells you whether a tool recognises a pattern someone planted, while a historical bug tells you whether it can reconstruct a conclusion that real engineers took real time to reach.

We chose Prefect because its history is public and auditable, and we chose PR #22761 because it is a clean candidate: one file, one well-isolated validation fix, reachable by calling a single function with a single bad argument. No scheduler internals, no worker pool, no timing window to win.

What does “sleep length must be non-negative” mean here?

It means a negative number reached Python’s time.sleep(), which rejects it at the interpreter level. Before the fix, Prefect’s pause and suspend functions accepted a timeout without validating it, and a negative value flowed quietly into an internal sleep call. The user’s mistake surfaced several frames away from the line that made it.

from prefect import flow
from prefect.flow_runs import pause_flow_run

@flow
def demo():
    pause_flow_run(timeout=-1)   # no validation: goes straight to time.sleep()

ValueError: sleep length must be non-negative
  # raised inside Prefect’s internals, not at the call site
  # file cited in the traceback: src/prefect/flow_runs.py

The fix Prefect shipped is the obvious one, stated up front rather than left to the interpreter:

if timeout is not None and timeout < 0:
    raise ValueError(f"'timeout' must be a non-negative number, got {timeout}")
context = FlowRunContext.get()

How do you reproduce a fixed bug on purpose?

Carefully, and on a branch you intend to throw away. The discipline that makes the result meaningful is that only the fix is reverted: everything else stays current, so the tool is reading today’s codebase with one historical defect in it.

• 1Cloned PrefectHQ/prefect and confirmed the fork carries genuine upstream commit history, not a hand-built fixture.
• 2Picked PR #22761 as a clean, single-file candidate reachable from one function call.
• 3Reverted only the guard lines in src/prefect/flow_runs.py, on a throwaway branch, leaving the rest of the tree untouched.
• 4Wrote a six-line flow that calls pause_flow_run(timeout=-1) from inside a real flow run, against a local Prefect server.
• 5Ran it. The bug reproduced for real: Prefect crashed with ValueError('sleep length must be non-negative') and wrote the full traceback into its own SQLite log store.
• 6Pointed the Tomosu Unified Runner at that log store in Prefect mode, letting it extract the failing file, fetch the still-buggy source from GitHub, and diagnose it through the same pipeline that runs in production.
• 7Only then compared Tomosu’s diagnosis and suggested fix against the real diff.

What did Tomosu find from the log alone?

Given the crash log and the repository, and nothing else, the analysis pinpointed pause_flow_run in src/prefect/flow_runs.py and stated the root cause as an unvalidated negative timeout reaching time.sleep(). It classified the finding under log failure signatures, rule LOG-01, at severity error, with a confidence of 74%, and proposed concrete code.

if timeout < 0:
    raise ValueError("timeout must be greater than or equal to 0")
if poll_interval <= 0:
    raise ValueError("poll_interval must be greater than 0")

How close was Tomosu’s fix to the real one?

Close on everything that decides whether the bug is understood, and imperfect on the edges. Here is the comparison line by line.

What does the one gap tell you?

That a suggested patch is a draft for a reviewer. timeout is optional in these functions, so None is a perfectly ordinary value; a bare if timeout < 0 would turn a valid call into a TypeError. A reviewer catches that in seconds. An automated merge would not.

Why does a real bug beat a synthetic benchmark?

Because a synthetic bug is written by someone who already knows what the right answer looks like, and that knowledge leaks into the test. The error message is tidy, the failing function is the one being examined, and nothing else in the repository is competing for attention. Real failures are not like that. This one surfaced several frames from the call that caused it, in a codebase of thousands of files, with a traceback that pointed at the victim rather than the culprit.

That is also why the comparison is worth publishing in full, gap included. A validation that reports only the parts that went well is a marketing exercise, and it tells a reader nothing about where to apply judgement.

How Tomosu does this on your repositories

The pipeline used here is the one that runs in production, not a special evaluation mode. In normal use it works the same way: a failure arrives in your logs, Tomosu extracts the failing path, reads the code that is actually implicated, and returns a diagnosis with a severity, a rule and a confidence, so a reviewer starts from a located root cause instead of a stack trace.

• →Connect a repository and get a baseline reliability score, free and on private code: see the repository scan guide or start a scan.
• →Put the same signal on the pull request, so a reliability verdict arrives before a merge rather than after an incident. That is the merge gate.
• →Give your coding agents the same view through the MCP server, so the code they write is checked against the same standard you review by.

Key takeaways

• Tomosu re-derived a real, already-fixed Prefect defect from a real crash log, with the known fix withheld until the final comparison.
• It identified the right file, the right function and the right root cause, and proposed a ValueError guard that solves the same problem as the shipped patch.
• Its patch missed one edge case, an optional timeout of None, and added a poll_interval check the real fix does not include.
• Treat a generated fix as a reviewer’s starting point. The located diagnosis is the part that saves the time.
• A historical bug from a public repository is a harder and more honest test than a synthetic one, because nobody wrote it to be found.
• One validated bug is a data point, not a hit rate. We publish the gap for the same reason we publish the result.

Frequently asked questions

A diagnosis is worth more than an alert. This validation ran against the open-source PrefectHQ/prefect repository, using the throwaway branch tomosu-eval-22761 to stage the reverted file. The branch was deleted afterwards and nothing was changed on Prefect’s main branch.

Discuss the validation →  ·  Questions: contact@tomosu.ai