Moved all the slow, flaky work onto a background worker fleet

Situation

Goodpods is a podcast app whose API had to keep about 400,000 podcasts up to date by scraping their RSS feeds, on top of compressing images, processing transcripts, and sending notifications for new episodes. Background jobs ran on a Celery setup tied into the old Django system, and the previous developer had to spend 30 minutes every day just managing its queues. Other expensive work still ran on the request path, so user actions that triggered it were slow.

Task

I took on building a replacement background-job subsystem from scratch, so that all slow, third-party-dependent work came off the request path and nobody inherited the daily queue babysitting.

Action

The first decision was Celery versus something new. We could have kept Celery, but it was entangled with the legacy Django system, so I chose RQ, a simple Python job queue backed by Redis, because it was more lightweight and easy to reason about. I started with a single example queue plus a monitoring dashboard, then moved image compression across with a 293-line test suite before touching anything critical. Next I built the core podcast refresh engine on a tiered schedule: podcasts with any subscribers refreshed every 2 hours, the rest every 4 days. I gave the worker fleet its own CI/CD pipeline so it deployed independently of the API, later switching to rolling updates so deploys did not interrupt running jobs. Two production lessons then shaped the design. First, user-triggered jobs like the push notification for tagging someone in a comment got stuck behind bulk work such as podcast refreshes and the weekly email, so I added priority queues to stop the starvation. Second, podcasts stopped updating when one bad feed could take down a refresh batch, which frustrated users and podcasters, so I added per-podcast error handling so a single broken feed could not kill the run. With everything migrated, I decommissioned the legacy Celery queues in November.

Result

Tagging someone in a comment went from taking 5 seconds to 80ms, and the same held for likes and anything else that notified another user. About 10 workers handled the load, with queue depths reaching 100,000 tasks during bulk jobs like the weekly email and recomputing user compatibility scores. The daily 30 minutes of queue management went away with Celery.