Metadata
- Author: Iavor Bojinov
- Full Title:: Want Your Company to Get Better at Experimentation?
- Category:: 🗞️Articles, Hypothesis Testing
- URL:: https://hbr.org/2025/01/want-your-company-to-get-better-at-experimentation
- Read date:: 2024-12-29
Highlights
many companies use online experimentation for just a handful of carefully selected projects. That’s because their data scientists are the only ones who can design, run, and analyze tests. It’s impossible to scale up that approach, and scaling matters. Research from Microsoft (replicated at other companies) reveals that teams and companies that run lots of tests outperform those that conduct just a few, for two reasons: Because most ideas have no positive impact, and it’s hard to predict which will succeed, companies must run lots of tests. And as the growth of AI—particularly generative AI—makes it cheaper and easier to create numerous digital product experiences, they must vastly increase the number of experiments they conduct—to hundreds or even thousands—to stay competitive. (View Highlight)
In an attempt to avoid negative results, they try to anticipate which ideas will have a big impact—something that is exceptionally difficult to predict. (View Highlight)
the findings they contain are never synthesized to identify patterns and generalizable lessons; nor are they archived in a standardized fashion. As a result, it’s not uncommon for different teams (or even the same team after its members have turned over) to repeatedly test an unsuccessful idea. (View Highlight)
both learning from individual experiments and learning across experiments to drive strategic choices on the basis of customer feedback. (View Highlight)
To achieve enterprise-wide experimentation for data-driven decisions, companies have to transition to a self-service approach (View Highlight)
No hay platform en el planeta que subsane la capacidad de razonar acerca de qué quieres testear y cómo vas a medir su éxito (y qué quieres controlar) Tampoco hay platform en el planeta que subsane que la navegación por tu producto sea inconsistente, que no haya cultura de medir bien… Ni la ensoñación de que en realidad no hace falta precisión en la medida porque si algo funciona “se va a notar”.
The data science organization (data scientists, data engineers, and software engineers) should ensure that the platform contains the following features, whether it is built internally or purchased. (View Highlight)
The ability to automatically impose statistical rigor. (View Highlight)
An AI assistant that provides easy-to-understand explanations of complex concepts. (View Highlight)
Once an organization is running hundreds or thousands of experiments a year, however, it becomes impossible to review every one of them in dedicated memos and meetings. Organizations should therefore shift their focus from analyzing individual experiments to analyzing, discussing, and learning from groups of related experiments, such as those concerning the search function or product-details pages that provide pictures, specifications, reviews, and other information. We refer to such efforts as experimentation programs. (View Highlight)
Learning across experiments at scale requires creating a knowledge repository—a system designed to store, categorize, and organize experiment results (including effects on sales and other key metrics, hypotheses about impacts on customers, and so on)—and making the information in it accessible to data scientists, product managers, and leadership. (View Highlight)