Cloud Run was exhausting our database connections, so I did the maths and fixed the root cause

Situation

Goodpods runs its API on Google Cloud Run, backed by a Cloud SQL Postgres database. In February 2025 we hit connection exhaustion in production. The user-facing impact was real: about 5% of requests were simply being dropped.

Task

I took on diagnosing and fixing the incident, from the immediate firefight through to the root-cause work.

Action

My first moves were mitigations to stop the bleeding. I lowered the connection pool size from 80 to 60 and cut pool_recycle from 3600 seconds to 300, so idle connections were returned much faster. I also added pre-ping, which checks a connection is alive before handing it out. The next day I removed pre-ping: it was filling the pools with idle connections, which wasted connections against the very limit we were hitting. Reverting it quickly mattered more than being right the first time.

That stabilised things, but the numbers said the real problem was architectural. At peak we ran 50 Cloud Run instances with 2 workers each, and each worker held its own pool. That meant a potential 160 connections per instance instead of 80, against a Cloud SQL maximum of 100. Autoscaling multiplied a config mistake into an outage.

So in May I fixed the arithmetic itself. I dropped the pool to 40 and simplified each instance to a single worker, halving the potential connection count. Then I moved read-heavy endpoints, like the feed algorithm, onto a read replica. Stale reads were not a concern because Cloud SQL replication is fast. I also split a noisy background task, compodability, onto its own queue so it could not starve API traffic.

Result

The potential connection count per instance was halved, bringing us back under the Cloud SQL limit with headroom. The heaviest read traffic no longer competed with writes for the primary’s connections at all. The fix addressed the actual cause, the workers-times-instances connection maths, rather than just turning knobs.