Built Detox E2E tests on CI, then removed them when they stopped paying off

Situation

In early 2021 I was working on the Goodpods React Native app. The team had just been reduced and we no longer had a full-time QA person, but we wanted to keep up the quality of releases. We wondered whether automated end-to-end testing could cover some of what manual QA had been doing.

Task

Trying Detox was a shared decision, but I took on the work of evaluating it and getting it running. My job was to wire end-to-end tests into our CI pipeline so the core flows were checked on every build.

Action

In February 2021 I set up Detox, an end-to-end testing framework for React Native that drives the real app on a device or simulator, and wired it into our Bitrise CI builds. I wrote specs covering the core account flows: signup, login, editing a profile, the profile page, and profile settings. The suite ran on CI for about eighteen months. In that time it caught a regression or two, but they were minor, and we were not keeping the tests up to date as the app changed. In September 2022, while modernising the native build setup, Detox broke and would have needed real effort to port across. Given the tests were already stale and had caught little, porting them was not worth it. Rather than leave a broken suite sitting in CI, I removed the whole Detox setup cleanly in one pass: the e2e specs, the configuration, and the dependency. We went back to covering releases with manual QA.

Result

Detox ran on our CI for roughly eighteen months and caught a couple of minor regressions in the account flows. When the build modernisation made its maintenance cost explicit, I made the call to delete my own earlier work rather than keep paying for tests we were not maintaining. The codebase came out of it with no dead test infrastructure, and quality checks continued through manual QA.