Understanding your A/B test results
Note: the information in this document only applies to testing campaign messages. See A/B Testing Newsletters for more on testing newsletters.
Chance to beat original (CTBO) is the likelihood that the variation will outperform the original.
It determines statistical significance by answering the question:
Is the difference observed between the original and variation greater than a difference due to random chance?
A CTBO of 50% means that the variation will outperform the original 50% of the time, which is the same as random chance. Therefore, there is no difference between your original and variation.
The closer your CTBO is to 50% (e.g., 40% or 60%), the less significant the difference between your original and variation.
The further your CTBO is from 50% (e.g., 5% and 90%), the more likely it is that there is a true difference between your original and variation.
Customer.io uses a significance level of 95%, meaning that CTBO has to be less than 5% or greater than 95% in order for us to report that one version is outperforming another.
If CTBO is > 95%, you can infer that your variation is outperforming your original.
If CTBO is < 5%, you can infer that your original is outperforming your variation.
If your CTBO is between 5% and 95%, it does not meet our threshold for statistical significance. If your sent volume is low, gathering more data may prove that a significant difference exists.
There are 3 steps to calculating CTBO:
1. Calculate the standard error for the original and variation.
x = number of messages with the desired behaviour (e.g. clicks) n = proportion of all messages with the desired behaviour (x / total, between 0 and 1) standardError = SQRT(( n * (1 - n) / x ))
2. Calculate the Z-score for the original and variation.
p_o = mean success rate for the original p_v = mean success rate for the variation se_o = standard error for the original se_v = standard error for the variation Z-score = (p_o-p_v)/SQRT(POWER(se_o,2)+POWER(se_v,2))
3. Based on the Z-score use a statistics table to determine the p-value and CTBO.
CTBO = 1 - p-value