Replaced FileZilla-to-prod with tests, pipelines and near-daily deploys

Situation

When I joined Goodpods the team was three developers: me on the API and two frontend devs. There was no test suite and no deploy pipeline. Changes were tested live on production by transferring files across with FileZilla. The whole app going down was a weekly occurrence, and deploys were scary because one wrong move would knock out production. Secrets either lived on one developer’s machine or were committed into the repo.

Task

Nobody assigned this to me; I took it on because the codebase could not be worked on safely in that state. I set out to build the testing, environment and deployment foundation for the team.

Action

I started with tests, writing a large suite for the legacy codebase partly to make sure I understood it all. Over July to October 2020 that suite grew to cover the main areas of the API: search, feeds, the player, podcasts, iTunes, notifications and recommendations. Alongside the tests I set up a dev environment so we could smoke test changes instead of trying them on production. Then I built the deploy pipeline itself: automatic deploys to UAT first, expanded shortly after into a full Dev/UAT/Prod topology. I moved all secrets out of source control and into environment files injected by CI. That was proactive hygiene rather than a response to a leak, but it closed off both bad patterns we had. Next I added SonarQube static analysis with a test coverage gate, and set the bar at 95%. That number was my call: high enough that we were not scared of breaking the legacy system, and the uncovered 5% was mostly boilerplate database connection code. I later tuned the coverage measurement to count only meaningful files, and the 95% target was relaxed once it had done its job. Finally I added automatic semantic version bumping on each release.

Result

Deploys went from roughly weekly, ad hoc and nerve-racking, to almost daily. When a release did go wrong, reverting was easy, and we did use that. The weekly incidents of the whole app going down stopped once proper CI/CD and dev environments were in place.