Metadata
- URL: https://notes.causal.engineering/archive/locally-optimal/
- Published Date: 2022-05-21
Highlights
- I have often argued that one of the most valuable types of data at a company is from A/B tests.
- often the effects are subtle enough that we need to be careful in deciding whether they are improvements.
- asked him why he hadn’t spent any energy on statistical problems at all and he had a great answer: “when a project we work on succeeds, we don’t need statistics to know it.”
- you can’t A/B test your way from selling horses to selling cars.
- One great point I’ve seen Josh Wills make repeatedly is that data doesn’t really matter until things go wrong.
Metadata
- Author: causal.engineering
- Full Title:: Locally Optimal
- Category:: 🗞️Articles
- Document Tags:: Data Team Vision and Mission,
- URL:: https://notes.causal.engineering/archive/locally-optimal/?utm_source=substack&utm_medium=email
- Read date:: 2025-03-31
Highlights
I asked him why he hadn’t spent any energy on statistical problems at all and he had a great answer: “when a project we work on succeeds, we don’t need statistics to know it.” (View Highlight)
Sure, incrementally better decisions add up to a lot of value over time,2 but maybe we’re just stuck in a local optimum and getting many small changes right will never get us to where we want to go. A modification of the famous Henry Ford quote kind of works here: you can’t A/B test your way from selling horses to selling cars. And a corollary: if you’re testing a horse against a car, you definitely don’t need an A/B test. (View Highlight)
You can think of debugging a broken product as roughly inverting the order of the A/B test. We start by observing a negative effect (by comparing last week to this week), then we go and look for potential explanations. Andrew Gelman and Guido Imbens nicely frame this task as “causes of effects” and contrast it with “effects of causes” (what we usually study with tools like A/B testing). “Causes of effects” is far more challenging because we need to generate hypotheses about what could have caused the problem and test each one until we find a probable explanation. (View Highlight)