Metadata
- Author: Olga Berezovsky
- Full Title:: Frequentist vs. Bayesian: Which Method Should You Choose for Your a/B Testing? - Issue 224
- Category:: 🗞️Articles
- URL:: https://dataanalysis.substack.com/p/frequentist-vs-bayesian-which-method?triedRedirect=true
- Finished date:: 2024-10-05
Highlights
Bayesian Method Cons: • Automating test analysis using the Bayesian approach is challenging. It’s easier to use tools that handle this rather than to develop your own. • Using a flat or flawed prior (inaccurate prior knowledge) can lead to incorrect results, especially with “data peaking.” • The lack of maximum sample size. While there are best practices for A/B testing with small data, not all teams follow them correctly. This can introduce errors or biases in Bayesian analyses. Follow my guide on How To Run An A/B Testing On Low Traffic. • More than 90% of tests are inconclusive. Bayesian method lacks the guidelines to make a decision regarding the test when the results are inconclusive. “In most cases, the data will indicate indecision, and we will not be sure what to do next. Should we continue the experiment and collect more data? Or go with the more probable variant even if the results are not conclusive?” from Bell Statistics. (View Highlight)