## Metadata - Authors: [[srivas-chennu|Srivas Chennu]] [[andrew-maher|Andrew Maher]] [[christian-pangerl|Christian Pangerl]] [[subash-prabanantham|Subash Prabanantham]] [[jae-hyeon-bae|Jae Hyeon Bae]] [[jamie-martin|Jamie Martin]] [[bu...|Bu..]] - Full Title:: Rapid and Scalable Bayesian AB Testing - Category:: #🗞️Articles - URL:: https://readwise.io/reader/document_raw_content/194828885 - Finished date:: [[2024-07-16]] ## Highlights > Statistical power in large-scale tests – as the number > of categorical factors and possible values of these factors grows, the amount of traffic allocated to each combination of values reduces. Hence a t-test run separately for each such combination would suffer from reduced statistical power, and consequently require the test to be run for a longer duration before true differences can be detected ([View Highlight](https://read.readwise.io/read/01j2w3ms1nrsf2d85wec9zfqrj)) > Hierarchical Bayesian inference is a well-established methodology previously articulated by Gelman and others [7], [8]. ([View Highlight](https://read.readwise.io/read/01j2w41a1ykbykksrr44txxagv))  ## Metadata - Authors: [[srivas-chennu|Srivas Chennu]] [[andrew-maher|Andrew Maher]] [[christian-pangerl|Christian Pangerl]] [[subash-prabanantham|Subash Prabanantham]] [[jae-hyeon-bae|Jae Hyeon Bae]] [[jamie-martin|Jamie Martin]] [[bu...|Bu..]] - Full Title:: Rapid and Scalable Bayesian AB Testing - Category:: #🗞️Articles - Document Tags:: [[Bayesian testing|Bayesian testing]], [[priority|Priority]], - URL:: https://arxiv.org/pdf/2307.14628 - Read date:: [[2025-03-23]] ## Highlights > the true but unknown probability of > a response is represented as rf, where the vector f = (m, c) is an element of F = M× C and denotes a specific content m presented to the user with a particular context c. Given a total of F = M + C content and context factors, each with cardinality of at most N, the number of unique f grows exponentially as O(NF). One of the statistical challenges with multivariate AB testing is that, as N and F grow, it becomes increasingly challenging to estimate each rf with sufficient statistical power. Our hierarchical Bayesian approach ameliorates this challenge, by pooling knowledge across different instances of f to increase statistical power. ([View Highlight](https://read.readwise.io/read/01jq0zc4k78nd4qm8jfhs1z4r5)) > we use a Bayesian hypothesis testing framework > that builds on the mSPRT [4]. The framework sequentially evaluates the relative evidence that there is a statistically significant difference between pairs of Bayesian estimates ˆrh > f s > – all the while maintaining statistical validity without the need for post hoc corrections. ([View Highlight](https://read.readwise.io/read/01jq0zeej7v42ss2e75b5aerzc)) > Another perspective is that experiments can be characterised by a set of common features. Within this perspective, pooling information and learnings across experiments can help experimenters build robust intuition as to the sorts of features that yield most impactful experiments. ([View Highlight](https://read.readwise.io/read/01jq0zfghw2gke80ng4edzkkhz)) > Hierarchical Bayesian inference, in addition to being useful > for modelling the variables within an experiment, can also be used to conduct such meta-analysis across experiments. By encoding distributional assumptions, we can learn the latent hyperparameters that can explain the impact of different experimental interventions. ([View Highlight](https://read.readwise.io/read/01jq0zfxh1vc13yctjq68s7w4y))