The article discusses scaling A/B testing infrastructure to handle multiple concurrent experiments. It highlights challenges faced when scaling from 10 to 100 experiments, including noisy results and pipeline debugging issues. The author shares insights gained from transitioning from a cron job-based setup to a more robust experimentation platform.
From the article
A very common question in any experimentation phase that comes sooner or later is “How to scale the A/B testing infrastructure from 10 to 100 concurrent experiments?” I remember the exact week our experimentation platform hit its first wall. We had 12 concurrent experiments running, a product team hungry for 10x that velocity, and a data infrastructure that was held together with scheduled cron jobs and a lot of quiet optimism. The results were getting noisy. Teams were stepping on each other’s experiments. And I was spending more time debugging pipelines than actually learning anything useful from the tests. That experience taught me more about experimentation infrastructure than any...
Skills
- statistical-analysis
- product-management
- ab-testing
- product-experimentation
