Diagnosing Connection Pool Exhaustion Between Cloud Run and PostgreSQL

At Acumen Academy I run a WhatsApp-based course delivery platform: Node.js and Express on Google Cloud Run, PostgreSQL behind it, Twilio handling the messaging. It delivers course content to learners around the world, and delivery comes in bursts, because course messages go out to whole cohorts at once.

This is the story of a production issue that only ever appeared during those bursts, and that turned out to be a design flaw rather than a bug. I am writing it up because the trap is built into the combination of any autoscaling platform and any per-instance connection pool, and I have since seen other teams walk into it.

The symptom: a database that was only sometimes full

Under normal traffic, everything was healthy. During delivery spikes, requests started failing with connection errors from PostgreSQL, the kind that look like this:

FATAL: remaining connection slots are reserved for non-replication superuser connections

The confusing part was that nothing was leaking. Connection counts always came back down after the spike. The database was well within capacity the rest of the day. Every individual component looked correctly configured, which is exactly why it took real diagnosis rather than a config skim to find.

The cause: pools multiply, limits don't

Here is the trap. A connection pool is configured per process. You pick a sensible pool size, say 10, and it feels safe because 10 is far below the database's connection limit.

But Cloud Run's whole job is to multiply your process. When a burst of traffic arrives, it scales out, and every new instance arrives holding its own pool of 10. The database's max_connections is a global limit, and it does not scale with anything.

So the real ceiling is:

instances × pool size ≤ max_connections (minus headroom)

With a pool of 10 and a limit of 100 connections, instance number eleven is the one that breaks you. And the painful irony is that autoscaling was working exactly as designed. The platform saw load and added capacity; the capacity is what took the database down.

This is why the failure only appeared during cohort sends. That was the only time traffic justified enough instances to cross the line.

How the diagnosis actually went

Two views of the system, laid side by side, made the cause obvious:

  1. Instance count over time from Cloud Run's metrics, plotted against the timestamps of the connection errors. The errors began precisely when instance count crossed a particular level, every single time.
  2. pg_stat_activity on the Postgres side during a spike, which showed the connection slots filled with healthy, idle connections from many different client addresses. Nothing stuck, nothing leaking. Just too many well-behaved pools.

That second observation is the tell that separates this problem from a connection leak. A leak shows old connections that never die. Pool multiplication shows fresh, tidy connections that are simply too numerous.

The fix, in layers

No single change fixes this properly, because the failure is an inequality with terms on both sides. We changed both sides:

  • Shrink the per-instance pool. Each Cloud Run instance handles a bounded number of concurrent requests, so a large pool per instance is mostly wasted anyway. The pool size should be derived from per-instance concurrency, not picked because it feels roomy.
  • Cap max instances. Cloud Run lets you set an upper bound on scaling. We set it so that the worst case of the inequality above stays below max_connections, with headroom reserved for migrations, admin sessions, and anything else that legitimately needs a slot.
  • Let idle pools shrink. Shorter idle timeouts mean instances that survive a burst release connections quickly instead of squatting on them until the next one.

For systems that need more concurrency than that arithmetic allows, the next step is server-side pooling with something like PgBouncer, which multiplexes many client connections onto few database connections. We did not need it once the numbers were right, but it is the correct escape hatch when capping instances is not acceptable.

The takeaway

In an autoscaling environment, the unit of connection budgeting is the fleet, not the process. Any per-process resource limit you set is silently multiplied by a number the platform chooses at the worst possible moment. It is worth doing this arithmetic for every pooled resource you have, before a busy day does it for you.