Holdout testing
How it works
A holdout is a type of A/B or random cohort test for email messages. But, rather than testing how your audience responds to variations of a message, some members of your audience will receive a message, and some won’t—they’re purposely “held out” of the test. This helps you determine whether a message is useful or not.
When you setup a holdout test in your campaign, you’ll determine the percentage of your audience you want to prevent from receiving a message. We’ll still generate deliveriesThe instance of a message sent to a person. When you set up a message, you determine an audience for your message. Each individual “send”—the version of a message sent to a single member of your audience—is a delivery. for your holdout group, but we send them to an internal message trap rather than sending them to your audience.
This lets us calculate conversions, so you can compare conversionsA campaign goal that you want your audience to achieve. You measure this goal as an attribute change or an event that occurs after a person receives a message within a campaign workflow. for people who receive a message to conversions for people who don’t, and determine the true utility of your message.
a/b test] --> b{email or hold out?} b-->c(people receive
messages) b--->|Holdout group
doesn't get a message|e(calculate
success rate) c-->f(calculate
success rate) e-->g{is there a
clear winner?} f-->g g-->|yes|h(choose
the winner) g-.->|no|i(wait for more data
or end the test)
Create a holdout test
It tends to be easier to set up an A/B test where one variant is the holdout. But, if you want to test multiple variants of a message, you can also perform holdout tests with a random cohort branch.
Either variation in your A/B test—email A or B—can be the holdout. But, for this example, we’re treating variation B as the holdout.
In your campaign’s Workflow, select the email you want to convert to a holdout test.
Click Turn into A/B test.
Set the amount of traffic that you want to send to your message (variation A) and holdout group (variation B).
Click Edit Variation B, and select Make this message a holdout test. This converts the “message” to a holdout, ensuring that it’s sent internally to a message trap; it is not sent or visible to your audience.
Check your sending behavior
The Sending Behavior has to be set to Queue Draft or Send Automatically for your holdout test to work because the holdout test “sends messages” to the message trap. If you use the Don’t Send setting, your holdout test won’t work.
Now, when you start your campaign, some people will receive Message A, and some people won’t get a message at all.
Random cohort holdout testing
If you’ve set up your own holdout test by creating a “black hole” address, or you want to test more than the two variants supported by an A/B test, you can use a random cohort branch to perform a holdout test.
Drag a Random Cohort Branch into your campaign or broadcast workflow.
Click the Random Cohort Branch action, set the percentage of people who will flow down each path, and click Save. Click Add Path if you want more than two branches in your random cohort.
Add messages to each branch in the test, and set up your message content for the “live” message. You don’t need to write content for the “holdout” variant, because it won’t be sent to your audience.
Click the message that you want to represent your holdout, and select Make this message a holdout test. This converts the “message” to a holdout, ensuring that it’s sent internally to a message trap; it is not sent or visible to your audience.
Now, when you start your campaign, a percentage of people will flow through your random cohort branch to the holdout variant. These people won’t receive a message.
Check holdout test performance
To check your results, go to your campaign and click the A/B Test tab. Here, you can select the “winner” of the test. The winner remains in your campaign, and the other message is removed. If your winner is the holdout, the message action is removed from your workflow entirely.
You can determine the winner using the Chance To Beat Original (CTBO) metric, in three different statistics: open rate, tracked link clicks, and conversion rate. You can learn more about how we calculate CTBO here.
In a holdout test, you should probably set a conversion goal and use the conversion metric to determine the true utility of your message. We don’t send a “holdout” message, so it’ll never be opened or clicked; the message variant that you send will always win those metrics. But we can calculate conversions for both the message and the holdout, making conversion goals—whether or not your message influences the behavior you want your audience to perform—the true measure of your message’s value.
If there isn’t a clear winner in your test result categories, we’ll show a “Not significant, need more data” message. You might need to let your test run longer to gather more results. If your test has already run for a while, and you’re confident that enough people have gone through the test, there might not be a statistically significant difference between your message and the holdout.
Holdout message status and metrics
Messages that are held out are “sent” and “delivered” to a message trap in Customer.io. We do this purely to help you visualize the performance differences between your real messages and your “held out” messages. But these messages never leave Customer.io.
Because holdout messages never leave Customer.io, they don’t affect your email domain’s deliverability ratings.
When you check the Sent tab from your campaign, holdout messages show their status as Holdout, so you can differentiate between your real messages from your holdout tests.
