How we raised mobile end-to-end test stability to 98%

A core part of Shopify’s DNA is that we make sure to foster a QA culture. For our mobile apps, that culture depends on a small layer of end-to-end (E2E) tests at the top of the test pyramid: flows that drive the real app the way a merchant would, running as blocking CI on every pull request. The pyramid works really well, as long as the suite is trustworthy.

The Shopify mobile app (our largest) had gotten to the point where it was blocking more good PRs than bad ones. Tests were getting flaky because screens might take an extra second to load and it got so bad that we had to pull the E2E suite from our PR checks entirely.

This is the story of how we fixed it, hitting 98% test stability—up from 50% using the old API.

The problem with our old setup