A memory leak users found first

In June 2018 a Celery user reported that a long-running event monitor leaked memory without bound. The fix was merged in August 2026, more than eight years later. We rewound Celery to the commit just before that fix, removed every clue, and ran a blind Tomosu scan. Tomosu found the exact failure mode and recommended the same fix the maintainers eventually shipped.

June 2018: a user reports real impact

Celery is one of the most widely used task queues in Python. Its real-time event monitoring keeps an in-memory view of tasks as they finish and as child tasks arrive, inside a process that runs continuously. Long-lived in-memory state like this makes retention mistakes expensive in production.

On June 21, 2018, a user running Celery 4.2.0 opened issue #4832:

The reporter had already traced the growth to Celery's event-state bookkeeping. They expected pending entries to be popped eventually. Instead, _tasks_to_resolve kept growing. They even asked whether it was "actually a problem in the celery code". It was.

How the leak works

When a child task's event arrives before its parent is known, Celery records the relationship so it can link them later. Nothing guaranteed that record would ever be removed.

1. A child task references a parent
2. The parent isn't currently known
3. Celery remembers the relationship
4. The parent may never arrive again
5. Tracking keys keep accumulating

No guaranteed cleanup plus no hard bound means memory can grow without limit. In a monitor that runs for weeks, that is an outage waiting to happen.

2026: the fix was tiny

Maintainers merged PR #10534, "Bound State._tasks_to_resolve to fix unbounded memory growth", on August 31, 2026. The PR confirms that unresolved-parent entries could stay forever, so a long-running monitor leaked memory without bound. The change itself is one line:

self._tasks_to_resolve = {}

self._tasks_to_resolve = LRUCache(max_tasks_in_memory)

The important part of the fix wasn't "use an LRU". It was the rule underneath it: this long-lived state must be bounded.

Why users found it first

This is the traditional reliability feedback loop. The defect ships, users feel it, and only then does anyone investigate.

• Code
• →
• Ship
• →
• User impact
• →
• Bug report
• →
• Investigate
• →
• Fix

Production became the place where the missing reliability invariant was discovered.

The experiment: a blind scan of the pre-fix code

We recreated celery/events/state.py exactly as it existed at commit fda3c5b07, immediately before the substantive fix: one file, 730 lines, about 28 KB. Then we removed the answer key.

• No bug report
• No PR description
• No patch
• No regression test
• No memory-leak hint

Tomosu saw the vulnerable source and nothing else. Not our diagnosis.

The result: Tomosu found the exact failure mode

Error · Reliability

In Memory Cluster State Scalability

“Events referencing never-seen parents can accumulate indefinitely.”

Tomosu recommended bounding the pending parent-child resolution index using max_tasks_in_memory.

Scan celery-state-prefix, September 7, 2026, finding #7.

Tomosu wasn't guessing "memory leak". It described the whole causal chain the eventual fix addresses: the mechanism (never-seen parents accumulate), the consequence (memory grows indefinitely) and the invariant (bound the unresolved state).

Tomosu vs. the eventual Celery fix

Same reliability invariant. Different implementation.

What we can actually claim

Tomosu identified this specific reliability defect in Celery's historical vulnerable code without being given the bug report or the eventual fix. That is a stronger statement than saying Tomosu "might have helped". The goal isn't perfect prediction. It is finding more reliability defects before customers become the detector.

• Code
• →
• Reliability analysis
• →
• Risk found
• →
• Engineer reviews
• →
• Ship

Discovery moves left of user impact.

The lesson is bigger than Celery

Every long-lived collection in a service needs an answer to four questions:

If the answer to the last two is "no" and "none", production owns the risk. Tomosu checks for exactly this kind of missing invariant on every change, so it can be fixed in the pull request instead of in an incident.

A user found it in 2018. Tomosu found it in the code. Run a free PRI scan to find reliability problems in your repository before your users do.