Chance to beat original (CTBO) is the likelihood that the variation will outperform the original.

It determines statistical significance by answering the question:

Is the difference observed between the original and variation greater than a difference due to random chance?

What does a CTBO of 50% mean?

A CTBO of 50% means that the variation will outperform the original 50% of the time, which is the same as random chance. Therefore, there is no difference between your original and variation.

The closer your CTBO is to 50% (e.g., 40% or 60%), the less significant the difference between your original and variation.

The further your CTBO is from 50% (e.g., 5% and 90%), the more likely it is that there is a true difference between your original and variation.

What does “Original is beating variation” or vice versa mean?

Customer.io uses a significance level of 95%, meaning that CTBO has to be less than 5% or greater than 95% in order for us to report that one version is outperforming another.

If CTBO is > 95%, you can infer that your variation is outperforming your original.

If CTBO is < 5%, you can infer that your original is outperforming your variation.

What does “Not significant, need more data” mean?

If your CTBO is between 5% and 95%, it does not meet our threshold for statistical significance. If your sent volume is low, gathering more data may prove that a significant difference exists.

How is ‘Chance to Beat Original’ CTBO calculated?

There are 3 steps to calculating CTBO:

1. Calculate the standard error for the original and variation.

2. Calculate the Z-score for the original and variation.

3. Based on the Z-score use a statistics table to determine the p-value and CTBO.