Scaling Reliable Experimentation in a Two-Sided AdTech Marketplace: ZMS Budget Split

Retail media has transformed e-commerce by allowing brands to advertise their products directly to consumers right at the digital point of sale. However, in a complex advertising marketplace, testing a new feature requires more than a conventional user-level A/B test.

Most A/B tests rest on a simple assumption: what User A does doesn't affect User B. In a standard consumer app, that's mostly true: you randomize users, show them different variants, and compare the results.

Retail media, however, operates slightly differently. It's a two-sided marketplace where advertisers compete for visibility via ad placements, customers see the results, and the whole system runs on a finite resource: advertisers' budgets. When experiment variants draw from the same campaign budgets, the statistical validity of the test is compromised. The control and treatment groups lose their independence, entering a cycle where they actively engage in Cannibalization Bias against one another.

In this blog post, we describe how we used Budget Split and Orthogonal Concurrency to address this challenge within our experimentation framework, and how these capabilities improved our testing between 2023 and 2025.