Chance to beat original (CTBO) is the likelihood that the variation will outperform the original.
It determines statistical significance by answering the question:
Is the difference observed between the original and variation greater than a difference due to random chance?
A CTBO of 50% means that the variation will outperform the original 50% of the time, which is the same as random chance. Therefore, there is no difference between your original and variation.
The closer your CTBO is to 50% (e.g., 40% or 60%), the less significant the difference between your original and variation.
The further your CTBO is from 50% (e.g., 5% and 90%), the more likely it is that there is a true difference between your original and variation.
Customer.io uses a significance level of 95%, meaning that CTBO has to be less than 5% or greater than 95% in order for us to report that one version is outperforming another.
If CTBO is > 95%, you can infer that your variation is outperforming your original.
If CTBO is < 5%, you can infer that your original is outperforming your variation.
If your CTBO is between 5% and 95%, it does not meet our threshold for statistical significance. If your sent volume is low, gathering more data may prove that a significant difference exists.
There are 3 steps to calculating CTBO:
1. Calculate the standard error for the original and variation.
2. Calculate the Z-score for the original and variation.
3. Based on the Z-score use a statistics table to determine the p-value and CTBO.