## Metadata - Author: [[tristan-handy|Tristan Handy]] - Full Title:: Down With Experimentation Maximalism! - by Tristan Handy Down With Experimentation Maximalism! - Category:: #🗞️Articles - URL:: https://roundup.getdbt.com/p/down-with-experimentation-maximalism - Read date:: [[2025-03-25]] ## Highlights > The assertion that the post from above—titled “[Locally Optimal](https://notes.causal.engineering/archive/locally-optimal/)”—makes is that structured experimentation programs are good for exploiting current product innovations (finding a local optimum), but are not helpful for identifying new product innovations. > > Sure, incrementally better decisions add up to a lot of value over time, but maybe we’re just stuck in a [local optimum](https://en.wikipedia.org/wiki/Local_optimum?utm_source=seanjtaylor&utm_medium=email&utm_campaign=locally-optimal) and getting many small changes right will never get us to where we want to go. A modification of the [famous Henry Ford quote](https://hbr.org/2011/08/henry-ford-never-said-the-fast?utm_source=seanjtaylor&utm_medium=email&utm_campaign=locally-optimal) kind of works here: you can’t A/B test your way from selling horses to selling cars. And a corollary: if you’re testing a horse against a car, you definitely don’t need an A/B test. ([View Highlight](https://read.readwise.io/read/01jq5x2mqpv479p6y9vvp5gjm0)) This is a match better expression of the idea I had regarding [[Hypothesis testing|Hypothesis Testing]] at [[freepik|Freepik]] about whether we win the match by big swings vs small increments. > *understand the context that your organization is operating in*. > • If the organization you’re working in is pre-PMF, you should be spending a lot of time generating hypotheses about what’s not yet working. This will likely look more like descriptive statistics, a lot of non-linear thinking, and collaborative problem-solving. You should resist the urge to rely on structured experimentation and instead be searching for insights that will unlock fundamentally new experiences for your customers. > • If you’re working in a post-PMF org, expect to spend more time inside of a structured experimentation program that is designed to optimize a process that is already working. There will be more guardrails but you’ll also likely get to use more sophisticated methods. ([View Highlight](https://read.readwise.io/read/01jq5x3fc512rysnctyr9m674k)) > I’m not suggesting that there are zero valid use cases for experimentation in a pre-PMF organization. However, I do think that experimentation has become a shibboleth within the data community, as if a structured experimentation program is *always* the gold standard for how innovation is done. ([View Highlight](https://read.readwise.io/read/01jq5x7rxce1hdexb937n3shq6))