# Understanding your A/B test results

Note: the information in this document only applies to testing campaign messages. See A/B Testing Newsletters for more on testing newsletters.

## What does ‘Chance to Beat Original’ mean?

Chance to beat original (CTBO) is the likelihood that the variation will outperform the original.

It determines statistical significance by answering the question:

Is the difference observed between the original and variation greater than a difference due to random chance?

### What does a CTBO of 50% mean?

A CTBO of 50% means that the variation will outperform the original 50% of the time, which is the same as random chance. Therefore, there is no difference between your original and variation.

The closer your CTBO is to 50% (e.g., 40% or 60%), the less significant the difference between your original and variation.

The further your CTBO is from 50% (e.g., 5% and 90%), the more likely it is that there is a true difference between your original and variation.

### What does “Original is beating variation” or vice versa mean?

Customer.io uses a significance level of 95%, meaning that CTBO has to be less than 5% or greater than 95% in order for us to report that one version is outperforming another.

If CTBO is > 95%, you can infer that your variation is outperforming your original.

If CTBO is < 5%, you can infer that your original is outperforming your variation.

### What does “Not significant, need more data” mean?

If your CTBO is between 5% and 95%, it does not meet our threshold for statistical significance. If your sent volume is low, gathering more data may prove that a significant difference exists.

## How is ‘Chance to Beat Original’ CTBO calculated?

There are 3 steps to calculating CTBO:

1. Calculate the standard error for the original and variation.

``````x = number of messages with the desired behaviour (e.g. clicks)
n = proportion of all messages with the desired behaviour (x / total, between 0 and 1)
standardError = SQRT(( n * (1 - n) / x ))
``````

2. Calculate the Z-score for the original and variation.

``````p_o = mean success rate for the original
p_v = mean success rate for the variation
se_o = standard error for the original
se_v = standard error for the variation
Z-score = (p_o-p_v)/SQRT(POWER(se_o,2)+POWER(se_v,2))
``````

3. Based on the Z-score use a statistics table to determine the p-value and CTBO.

``````CTBO = 1 - p-value
``````
Copied to clipboard!  