Bayesian experimentation

Learn how Bayesian A/B testing works and how to interpret probabilities, credible intervals, and e-values.

Algolia A/B testing uses Bayesian statistics to help you read your results. This page covers how the Bayesian approach works, how to read its output, and how it differs from the frequentist approach.

How Bayesian experimentation works

A Bayesian test combines what it sees in your data with what you knew before the test began. It asks one question: “Given the data so far, what’s the chance that variant B beats variant A?”. A frequentist test asks something else: “How odd is this data if the variants are in fact the same?”

• A Bayesian test starts with a prior, a probability distribution that represents assumptions about a metric before the test begins
• Algolia combines the prior with the test’s data to calculate the posterior, an updated probability distribution that reflects the observed results.

This is why you can look at a Bayesian test whenever you want. Each check shows you the current estimate of which variant leads. You don’t have to wait for a set number of users, and a mid-test check doesn’t break the math.

How to interpret Bayesian results

Bayesian probabilities and “credible intervals” describe uncertainty under the test’s model and “prior”. Your test results include:

• Probability to be better. For example, “Variant B has a 95% probability of beating the control.” Algolia recommends a probability of at least 95% when evaluating whether a variant outperforms the control.
• Credible interval. A range that holds the true value of a metric, with a stated chance. For example, “There’s a 95% chance that the true uplift falls between 1.2% and 4.8%.” That metric is often the gap in conversion rate between the variants. You can’t read a frequentist confidence interval this way.
• Evidence status. Algolia computes the evidence status for the primary metric you selected when creating the test. An evidence status of “enough” means the test has gathered enough data. For Bayesian tests, this status rests on e-values. The bar is an e-value of 20 or more.

An e-value weighs your data against the case for no real difference. It updates as data arrives, and it holds up no matter how often you check it. That’s what makes it safe to watch a test as it runs. Higher is stronger: an e-value of 20 means the data fits a real difference 20 times better than no difference.

Below is an example of the evidence status in the A/B test overview page. To learn more, see Safe testing.

A higher probability to be better makes a stronger case for a variant. Since the numbers change with each batch of data, you can watch a trend build instead of judging each check on its own. For the other things to weigh before you act, see How to read your A/B test results.

Benefits of Bayesian experimentation

• Direct answers. A probability that “variant B beats the control” answers the business question as asked.
• Frequent checks, no added risk. Fixed-duration frequentist tests can suffer the “peeking problem”: stopping when interim results become significant can increase the risk of a false positive. Algolia’s Bayesian tests account for repeated checks, so you can check as often as you like.
• No set duration up front. For frequentist tests in Algolia, you need ti plan the test duration before launch. A Bayesian test lets you follow it and stop once the evidence is strong.

How Bayesian and frequentist testing compare

Both methods are sound ways to test an idea, but they answer different questions. Neither is the better one. Frequentist testing has a long track record and firm guarantees, but its rules are strict. Algolia’s Bayesian testing gives you more flexibility in when to stop and reports results as probabilities. That flexibility fits online A/B testing, where data keeps coming in and you keep iterating.

Aspect	Frequentist	Bayesian
Core question	How odd is this data, or data more extreme, if there’s no real effect?	Given the data, what’s the chance of each possible effect size?
Main output	p-value and confidence interval	Probability to be better, credible interval
Interpretation	Indirect: it tells you how odd your data looks if nothing changed	Direct: it tells you the chance that “B beats the control” is true
Checking results before the end	For fixed-duration tests, wait until the planned end before making a decision	Review results during the test and use the evidence status when deciding whether to stop
Sample size planning	Typically needs a sample size fixed in advance	Doesn’t need a fixed sample size, though more data sharpens the estimate
Prior distribution	Not used	Combined with observed data to calculate the posterior

For information about how Algolia computes frequentist confidence, see How A/B test scores are calculated.