## Metadata * URL: [https://notes.causal.engineering/archive/locally-optimal/](https://notes.causal.engineering/archive/locally-optimal/) * Published Date: 2022-05-21 ## Highlights * I have often argued that one of the most valuable types of data at a company is from A/B tests. * often the effects are subtle enough that we need to be careful in deciding whether they are improvements. * asked him why he hadn’t spent any energy on statistical problems at all and he had a great answer: “when a project we work on succeeds, we don’t need statistics to know it.” * you can’t A/B test your way from selling horses to selling cars. * One great point I’ve seen Josh Wills make repeatedly is that data doesn’t really matter until things go wrong. ## Metadata - Author: [[causal.engineering|Causal]] - Full Title:: Locally Optimal - Category:: #🗞️Articles - Document Tags:: [[Data team vision and mission|Data Team Vision And Mission]], - URL:: https://notes.causal.engineering/archive/locally-optimal/?utm_source=substack&utm_medium=email - Read date:: [[2025-03-31]] ## Highlights > I asked him why he hadn't spent any energy on statistical problems at all and he had a great answer: "when a project we work on succeeds, we don't need statistics to know it." ([View Highlight](https://read.readwise.io/read/01jqkytdq00p3kmdw1znsf7hk9)) > Sure, incrementally better decisions add up to a lot of value over time,[2](https://notes.causal.engineering/archive/locally-optimal/?utm_source=substack&utm_medium=email#fn:2) but maybe we're just stuck in a [local optimum](https://en.wikipedia.org/wiki/Local_optimum?utm_source=seanjtaylor&utm_medium=email&utm_campaign=locally-optimal) and getting many small changes right will never get us to where we want to go. A modification of the [famous Henry Ford quote](https://hbr.org/2011/08/henry-ford-never-said-the-fast?utm_source=seanjtaylor&utm_medium=email&utm_campaign=locally-optimal) kind of works here: you can't A/B test your way from selling horses to selling cars. And a corollary: if you're testing a horse against a car, you definitely don't need an A/B test. ([View Highlight](https://read.readwise.io/read/01jqkytzpzdxsm3mbnfd34ga3d)) > You can think of [debugging a broken product](https://en.wikipedia.org/wiki/Root_cause_analysis?utm_source=seanjtaylor&utm_medium=email&utm_campaign=locally-optimal) as roughly inverting the order of the A/B test. We start by observing a negative effect (by comparing last week to this week), then we go and look for potential explanations. Andrew Gelman and Guido Imbens [nicely frame](https://www.nber.org/papers/w19614?utm_source=seanjtaylor&utm_medium=email&utm_campaign=locally-optimal) this task as "causes of effects" and contrast it with "effects of causes" (what we usually study with tools like A/B testing). "Causes of effects" is far more challenging because we need to generate hypotheses about what could have caused the problem and test each one until we find a probable explanation. ([View Highlight](https://read.readwise.io/read/01jqm49htpxm4nyqh8tyc67aq8))