## Metadata - Author: [[evan-miller|Evan Miller]] - Full Title:: How Not to Run an a/B Test - Category:: #🗞️Articles - Document Tags:: [[Hypothesis testing|Hypothesis Testing]], - URL:: http://www.evanmiller.org/how-not-to-run-an-ab-test.html - Finished date:: [[2024-05-04]] ## Highlights > Since you are going to fix the sample size in advance, what sample size should you use? This formula is a good rule of thumb: > Where is the minimum effect you wish to detect and is the sample variance you expect. Of course you might not know the variance, but if it’s just a binomial proportion you’re calculating (e.g. a percent conversion rate) the variance is given by: ([View Highlight](https://read.readwise.io/read/01hx20f6hvn177vh7enaz2h0h1)) > **UPDATE, May 2013:** You can see this formula in action with my new interactive **[Sample Size Calculator](https://www.evanmiller.org/ab-testing/sample-size.html)**. Enter the effect size you wish to detect, set the power and significance levels, and you'll get an easy-to-read number telling you the sample size you need. **END OF UPDATE** ([View Highlight](https://read.readwise.io/read/01hx20ftayyj7kdmgq8dhcm4fs)) > report how large of an effect can be detected given the current sample size. ([View Highlight](https://read.readwise.io/read/01hx20g9by3wgbhqn8w4h74vgb))