Connection Leak vs. Pool Exhaustion: How to Tell the Difference

Every connection is checked out, requests are queuing, and the pool is throwing timeouts. The next decision depends on one question: is something leaking connections, or is the pool simply being asked for more than it can give? Those two problems produce the same error and need opposite fixes.

This guide is language-agnostic. The mechanism is the same whether the pool is HikariCP on the JVM, pg.Pool in Node.js, SQLAlchemy’s QueuePool in Python, or Go’s database/sql. Where the tools differ, there is a per-stack table. For a JVM-specific walkthrough of HikariCP’s exception, leak detection, and thread dumps, see HikariCP “Connection Is Not Available, Request Timed Out”: How to Find the Real Cause.

What is the difference between a connection leak and pool exhaustion?

Pool exhaustion is the condition where every connection a pool is allowed to hold is checked out, so the next caller has to wait, and eventually times out. It is a symptom. A connection leak is a specific bug: a code path checks out a connection and never returns it to the pool, so the pool’s usable capacity shrinks by one each time that path runs. A leak always ends in exhaustion if the process lives long enough. Exhaustion does not imply a leak.

The other causes of exhaustion return every connection eventually, but not fast enough:

• Load: request rate × normal hold time exceeds the pool size at peak.
• Slow queries: a missing index, a lock wait, or a bigger table stretches each hold. The production-only slow query is the most common version.
• Long holds in application code: a transaction that spans an HTTP call, a queue publish, or slow processing.
• Nested acquisition: a thread holding one connection waits for a second, which can deadlock the pool at modest concurrency.
• Amplified demand: retries from clients add requests exactly when each request is slowest. Retry storms exhaust pools this way.

Search Stack Overflow for pool exhaustion and most threads end with “increase the pool size” or “you have a leak”, often without evidence for either. So the practical question is not “leak or exhaustion”. It is “leak, or one of the causes that recover on their own”. The answer changes the fix completely: a leak needs a code change on a specific path, while the others need faster work, shorter holds, less demand, or (last) more connections.

How do a leak and exhaustion look different over time?

At the moment of the timeout, both look identical: in-use equals the maximum, idle is zero, callers are waiting. The difference is only visible as a time series. Put checked-out (in-use) connections on the same chart as request rate, over at least a full daily cycle, and the two causes separate.

Three details on that chart carry most of the diagnosis:

• Does in-use come down in the quiet hours? If traffic drops by 80% overnight and in-use does not move, those connections are not doing work for anyone.
• What does a restart do? A restart that instantly drops in-use to its baseline, followed by a slow climb, is the classic leak signature. If the pool is full again within minutes of a restart, look at load or slow work first.
• Where are the steps? Leaks step up when the leaking path runs. If the steps line up with a batch job, a retry burst, or error spikes on one endpoint, you have a strong lead on which path it is.

What does exhaustion look like in Java, Node.js, Python, and Go?

Every pool has a maximum size, a way to wait for a free connection, and some way to report how many connections are checked out. The defaults differ in ways that change what the incident looks like. In particular, two of the four common stacks do not fail fast by default.

Defaults as documented by HikariCP, node-postgres, SQLAlchemy, and Go. Frameworks often override them, so read the effective values from your running configuration.

Two of those defaults deserve a closer look because they change the symptom, not just the timing.

node-postgres waits forever by default

With connectionTimeoutMillis at its default of 0, a caller that cannot get a client waits with no limit. An exhausted pool does not throw. Requests just stop completing, which often surfaces as a load balancer or gateway timeout far from the real cause. Set a timeout so the failure is visible and attributable, and export the three counters the pool already exposes:

const pool = new Pool({
  max: 10,
  connectionTimeoutMillis: 5_000,   // default 0 means wait forever
  idleTimeoutMillis: 10_000,
});

setInterval(() => {
  metrics.gauge('db.pool.total', pool.totalCount);
  metrics.gauge('db.pool.idle', pool.idleCount);
  metrics.gauge('db.pool.waiting', pool.waitingCount);
  metrics.gauge('db.pool.in_use', pool.totalCount - pool.idleCount);
}, 10_000).unref();

Go’s database/sql is unlimited by default

With SetMaxOpenConns left at 0, a Go service never exhausts its own pool. A leak instead opens new connections until the database refuses them, for example PostgreSQL’s “sorry, too many clients already” or MySQL’s “Too many connections”. That error can hit every other service sharing the database, which makes a single service’s leak look like a platform-wide outage. Set a limit and log the stats:

db.SetMaxOpenConns(20)             // default 0 = unlimited
db.SetMaxIdleConns(20)             // default 2
db.SetConnMaxLifetime(30 * time.Minute)

go func() {
    t := time.NewTicker(15 * time.Second)
    defer t.Stop()
    for range t.C {
        s := db.Stats()
        log.Printf("db pool open=%d inuse=%d idle=%d waits=%d waited=%s",
            s.OpenConnections, s.InUse, s.Idle, s.WaitCount, s.WaitDuration)
    }
}()

On any stack, the four numbers that matter are the same: maximum, in use, idle, and waiting (or cumulative wait count and time). If you only export one extra metric after reading this, make it in-use as a time series.

How do I confirm a leak from the database side?

The pool knows which connections are checked out. The database knows what each of those connections is actually doing. Comparing the two views is the quickest confirmation of a leak, and it works on any stack.

SELECT application_name, state, count(*) AS sessions,
       max(now() - state_change) AS longest_in_state
FROM   pg_stat_activity
WHERE  datname = current_database()
  AND  backend_type = 'client backend'
GROUP BY 1, 2
ORDER BY 3 DESC;

The column meanings are in the PostgreSQL cumulative statistics documentation. Set application_name in each service’s connection string so you can filter one service’s pool. Then line the result up against the pool’s own counters at the same moment.

MySQL shows the same picture in SHOW FULL PROCESSLIST (Sleep with a large Time) and information_schema.innodb_trx.

Why does a leak show up during a different incident?

Most real leaks are not on the happy path. The happy path is exercised by every test and every request, so a missing release there is found within hours. Leaks survive on error and early-return paths: the branch taken when a row fails to parse, a downstream call throws, a validation check fails, or a request is cancelled. That has an important consequence: a leak drains the pool at the rate its path runs.

This is why “the pool ran out during the payment provider outage” is often a misdiagnosis. The provider outage was real. The pool exhaustion was a second, pre-existing bug that the outage exposed. The tell is what happens after the dependency recovers: exhaustion from load or slowness clears within seconds, while a leak leaves the pool full until someone restarts the service.

The same logic explains leaks that appear “right after a deploy”. The deploy may not have added the leak. It may have added a new failure mode, a stricter validation, or a new timeout that sends more traffic down an old error branch. When you look for the commit that caused the incident, include changes that increase the rate of existing error paths, not only changes to the data access code.

Which code patterns leak connections?

The patterns are the same across languages: acquisition without a guaranteed release, and resources that pin a connection (result sets, cursors, streams, transactions) that are not closed on every path.

Node.js: client.release() skipped when a query throws

async function transfer(from, to, amount) {
  const client = await pool.connect();
  await client.query('BEGIN');
  await client.query(DEBIT_SQL, [from, amount]);   // may throw
  await client.query(CREDIT_SQL, [to, amount]);
  await client.query('COMMIT');
  client.release();                                  // never reached on error
}

async function transfer(from, to, amount) {
  const client = await pool.connect();
  let broken;
  try {
    await client.query('BEGIN');
    await client.query(DEBIT_SQL, [from, amount]);
    await client.query(CREDIT_SQL, [to, amount]);
    await client.query('COMMIT');
  } catch (err) {
    await client.query('ROLLBACK').catch((e) => { broken = e; });
    throw err;
  } finally {
    client.release(broken);   // always runs; truthy arg destroys it
  }
}

For single statements, pool.query() checks out and releases a client for you, so it cannot leak. The leak risk lives in code that calls pool.connect() directly, usually for transactions.

Go: rows or tx left open on an early return

rows, err := db.QueryContext(ctx, listItemsSQL, orderID)
if err != nil {
    return nil, err
}
for rows.Next() {
    var it Item
    if err := rows.Scan(&it.ID, &it.SKU, &it.Qty); err != nil {
        return nil, err   // rows still open: its connection stays in use
    }
    items = append(items, it)
}
return items, rows.Err()

rows, err := db.QueryContext(ctx, listItemsSQL, orderID)
if err != nil {
    return nil, err
}
defer rows.Close()   // safe to call even after Next has returned false

tx, err := db.BeginTx(ctx, nil)
if err != nil {
    return err
}
defer tx.Rollback()  // no-op after a successful Commit

When rows.Next() returns false the rows are closed automatically, which is why the happy path in the broken example does not leak. Only the early return does, and only when a Scan fails. That is exactly the kind of path tests rarely exercise.

Python: a Session that is not closed on an exception

# Broken: close() is skipped if anything between here and there raises
def load_invoice(invoice_id):
    session = SessionLocal()
    invoice = session.get(Invoice, invoice_id)
    render_pdf(invoice)                 # raises: session keeps its connection
    session.close()

# Fixed: the context manager closes the Session on every path
def load_invoice(invoice_id):
    with SessionLocal() as session:
        invoice = session.get(Invoice, invoice_id)
        render_pdf(invoice)             # connection returned when the block exits

In CPython, reference counting sometimes returns such a connection when the abandoned object is garbage collected, and SQLAlchemy may log a warning when it has to clean up a connection that was never checked in. That makes Python leaks intermittent rather than absent. Do not rely on it; frameworks such as FastAPI dependencies with yield, or scoped_session.remove() at request end, exist to make release structural.

Other patterns that pin a connection

• Streams and cursors returned to the caller: a repository method that returns a lazy stream or server-side cursor keeps its connection until the caller closes it.
• Transactions left open by an early return between BEGIN and COMMIT, which also hold locks.
• Connections stored in fields or passed to another thread, where ownership of the release is unclear.
• Background jobs that open a connection outside the request lifecycle, so request-scoped cleanup never runs.

For the JVM versions of these (unclosed Stream<T> repository results, REQUIRES_NEW nesting, and open-session-in-view), see the HikariCP guide and the sibling post on resource leaks in Java services.

Track the borrower when your pool will not

HikariCP has leakDetectionThreshold. node-postgres does not have a built-in equivalent, but you can add one in a few lines by recording where each client was checked out:

const held = new Map();

async function connect() {
  const client = await pool.connect();
  held.set(client, { at: Date.now(), stack: new Error('checked out here').stack });
  const release = client.release;
  client.release = (err) => { held.delete(client); return release(err); };
  return client;
}

setInterval(() => {
  const now = Date.now();
  for (const { at, stack } of held.values()) {
    if (now - at > 30_000) console.warn(`db client held ${now - at} ms`, stack);
  }
}, 10_000).unref();

As with HikariCP, a warning means “held a long time”, not “leaked”. A warning for a client that is later released is a long hold. A warning that repeats from the same stack and never clears is a leak. In SQLAlchemy, the pool’s checkout and checkin events give you the same hook.

A step-by-step way to tell a leak from pool exhaustion

1. Read the error and the pool counters. Record maximum, in use, idle, and waiting at the moment of failure, and any chained cause. If total is below the maximum while callers wait, the pool is failing to create connections, which is a separate problem.
2. Chart in-use against request rate over a full day. Per instance, against instance uptime. A ratchet that ignores quiet periods and resets on restart points to a leak; a curve that follows traffic points to load or slow work.
3. Compare the pool’s view with the database’s. Many connections checked out in the pool but idle or long idle in transaction at the database confirms a leak. Mostly active sessions point to load or slow queries.
4. Turn on borrower tracking. Use the pool’s leak detection, or a wrapper that records the checkout stack and age. Keep warnings that never clear; they name the leaking path.
5. Find every acquisition path without structured release. Search for manual checkouts, transactions, returned cursors, and streams. Check the error and early-return branches, not just the happy path.
6. Fix, then verify the shape. After the fix ships, in-use should return to its baseline in quiet periods and after the next dependency failure. If it still ratchets, there is a second path.

Fixes, matched to the cause

Treat any change to pool size, pool timeout, or data access code as a change to a shared production resource. Several services and background jobs usually share the database’s connection budget, so a local “fix” can move the exhaustion somewhere else. Assessing the blast radius of the change is part of the fix.

How Tomosu helps

Runtime metrics tell you that the pool is leaking. They rarely tell you every path that can leak, because a path only shows up in a leak trace after it has run in production. Tomosu analyzes the repository and each pull request for this kind of production reliability risk, and for connection handling it focuses on the code:

• Acquisition paths without structured release: manual checkouts, transactions, and cursors where an error or early-return branch can skip the release.
• Holds that stretch: remote calls, retries, or slow work performed while a connection or transaction is open.
• Shared blast radius: which endpoints and jobs share the pool, so a finding on a rarely used admin path is weighed differently from one on checkout.
• Evidence to collect: the metrics and database queries that would confirm or rule out the finding, rather than a generic “increase the pool” suggestion.

These findings feed the Production Reliability Index, so a pull request that adds a pool.connect() without a finally is visible at review time rather than during the next dependency outage.

Scan your repository with Tomosu →

Key takeaways

• Pool exhaustion is a symptom. A connection leak is one cause of it; load, slow queries, long holds, and nested acquisition are others.
• Chart in-use connections against request rate, per instance. A leak ratchets and ignores quiet periods; exhaustion from load recovers.
• Compare the pool’s view with the database’s. Connections checked out but idle at the database are leak candidates.
• Most leaks live on error and early-return paths, so they drain the pool at the error rate and surface during other incidents.
• node-postgres waits forever and Go opens unlimited connections by default. Set explicit limits so exhaustion is visible.
• Fix leaks with structured release on every path. A bigger pool only delays a leak.

Frequently asked questions

A pool never runs out on its own. Some code path took a connection and did not give it back in time, and the fix lives on that path. Assess your repository →