Per-request monitoring at over 1M requests a day, without buying a monitoring product

Situation

Goodpods is a podcast app whose API handled over 1M requests a day. In January 2024 we were tasked with finding which endpoints were slow and fixing them, but we had no visibility into request performance. The client did not want to pay for another monitoring service; we had used Sentry in the past and dropped it.

Task

I took on building our own request monitoring, so we could answer the “which endpoints are slow” question with real data instead of guesses.

Action

I wrote middleware that recorded every request: URL, method, status code, timing, request and response sizes, the authenticated identity, and whether it was a cache hit or miss. Each record went into Redis, and a daily task aggregated the data, archived it, and posted alerts to Slack.

The first design had a problem. At over 1M requests a day, storing a full record per request gave the monitoring data a large footprint on our main Redis instance, the same instance the API depended on. After doing the maths on memory use, I decided key minimisation was important, so I cut every field name down to two letters (me for method, st for status, ts for timing, and so on). That kept the per-request cost small enough to run monitoring on every single request rather than sampling.

I also handled the cache-hit edge case, where requests served from cache still needed to be recorded correctly, wrote a test for the aggregation, and ran the daily task on our develop environment before turning it on in production.

Result

The monitoring let us calculate user misery metrics: which endpoints were causing users to wait the most. We targeted the top 10 of those endpoints for speed-ups, and most of them just needed better indexes or new compound indexes, indexes covering several columns at once so the database can satisfy the whole query from one lookup.