Holdout testing

How it works

A holdout is a type of A/B or random cohort test for email messages. But, rather than testing how your audience responds to variations of a message, some members of your audience will receive a message, and some won’t—they’re purposely “held out” of the test. This helps you determine whether a message is useful or not.

When you setup a holdout test in your campaign, you’ll determine the percentage of your audience you want to prevent from receiving a message. We’ll still generate deliveriesThe instance of a message sent to a person. When you set up a message, you determine an audience for your message. Each individual “send”—the version of a message sent to a single member of your audience—is a delivery. for your holdout group, but we send them to an internal message trap rather than sending them to your audience.

This lets us calculate conversions, so you can compare conversionsA campaign goal that you want your audience to achieve. You measure this goal as an attribute change or an event that occurs after a person receives a message within a campaign workflow. for people who receive a message to conversions for people who don’t, and determine the true utility of your message.

flowchart LR a[people enter
a/b test] --> b{email or hold out?} b-->c(people receive
messages) b--->|Holdout group
doesn't get a message|e(calculate
success rate) c-->f(calculate
success rate) e-->g{is there a
clear winner?} f-->g g-->|yes|h(choose
the winner) g-.->|no|i(wait for more data
or end the test)

Create a holdout test

It tends to be easier to set up an A/B test where one variant is the holdout. But, if you want to test multiple variants of a message, you can also perform holdout tests with a random cohort branch.

Either variation in your A/B test—email A or B—can be the holdout. But, for this example, we’re treating variation B as the holdout.

  1. In your campaign’s Workflow, select the email you want to convert to a holdout test.

  2. Click Turn into A/B test.

    image.png
    image.png

  3. Set the amount of traffic that you want to send to your message (variation A) and holdout group (variation B).

    Set the percentages of people who will receive a message and who won't
    Set the percentages of people who will receive a message and who won't

  4. Click Edit Variation B, and select Don’t deliver this message to people. This converts the “message” to a holdout, ensuring that it’s sent internally to a message trap; it is not sent or visible to your audience.

    Hold message b out of the test
    Hold message b out of the test
  5. Now, when you start your campaign, some people will receive Message A, and some people won’t get a message at all.

Random cohort holdout testing

If you’ve set up your own holdout test by creating a “black hole” address, or you want to test more than the two variants supported by an A/B test, you can use a random cohort branch to perform a holdout test.

  1. Drag a Random Cohort Branch into your campaign or broadcast workflow.

  2. Click the Random Cohort Branch action, set the percentage of people who will flow down each path, and click Save. Click Add Path if you want more than two branches in your random cohort.

    image.png
    image.png

  3. Add messages to each branch in the test, and set up your message content for the “live” message. You don’t need to write content for the “holdout” variant, because it won’t be sent to your audience.

  4. Click the message that you want to represent your holdout, and select Don’t deliver this message to people. This converts the “message” to a holdout, ensuring that it’s sent internally to a message trap; it is not sent or visible to your audience.

    Hold message b out of the random cohort
    Hold message b out of the random cohort

Now, when you start your campaign, a percentage of people will flow through your random cohort branch to the holdout variant. These people won’t receive a message.

Check holdout test performance

To check your results, go to your campaign and click the A/B Test tab. Here, you can select the “winner” of the test. The winner remains in your campaign, and the other message is removed. If your winner is the holdout, the message action is removed from your workflow entirely.

Determine the utility of your message in the A/B test tab
Determine the utility of your message in the A/B test tab

You can determine the winner using the Chance To Beat Original (CTBO) metric, in three different statistics: open rate, tracked link clicks, and conversion rate. You can learn more about how we calculate CTBO here.

In a holdout test, you should probably set a conversion goal and use the conversion metric to determine the true utility of your message. We don’t send a “holdout” message, so it’ll never be opened or clicked; the message variant that you send will always win those metrics. But we can calculate conversions for both the message and the holdout, making conversion goals—whether or not your message influences the behavior you want your audience to perform—the true measure of your message’s value.

If there isn’t a clear winner in your test result categories, we’ll show a “Not significant, need more data” message. You might need to let your test run longer to gather more results. If your test has already run for a while, and you’re confident that enough people have gone through the test, there might not be a statistically significant difference between your message and the holdout.

Holdout message status and metrics

Messages that are held out are “sent” and “delivered” to a message trap in Customer.io. We do this purely to help you visualize the performance differences between your real messages and your “held out” messages. But these messages never leave Customer.io.

Because holdout messages never leave Customer.io, they don’t affect your email domain’s deliverability ratings.

When you check the Sent tab from your campaign, holdout messages show their status as Holdout, so you can differentiate between your real messages from your holdout tests.

messages in a holdout test show their status as holdout
messages in a holdout test show their status as holdout
Copied to clipboard!
Is this page helpful?