Setting up Blackhole/Holdout Testing
A blackhole test (also known as a holdout test) is a special type of cohort testing: email vs. no email! You want to send some of your users an email, and others to receive nothing. This recipe will help you get that set up.
Note: If you’d like to test something other than content/no-content, such as timing or single email vs. multiple emails, we have a cohort testing recipe for you!
All you’ll need is:
- Access to your email admin (in order to create a new email address)
This is essentially an email address that you will ignore. Its purpose is to send some of your messages to an unmonitored address that you control instead of to your customers. This is usually named something like
The email address you choose should be a real email address that you set up on a domain you control but that you DO NOT set up in an email client such as Outlook or Gmail. Setting up a real email address for your blackhole email address will prevent the messages that are sent to it from bouncing due to an invalid email address. Also, NOT setting up an email client to check the blackhole address will prevent our tracking pixels from getting loaded and causing your blackhole messages to be marked as opened by the email client even if they have never been opened.
If you plan to send a large number of messages to this blackhole address, you may wish to configure the blackhole email address (at the server level) so that these messages are automatically deleted in order to avoid exceeding any storage limits your email provider may have.
Within your campaign, add an A/B test to the email template you would like to test. This is done by clicking the email item to expand the edit panel, and then clicking on the Turn into A/B test button to the right of the item.
The ‘blackhole’ part comes next, in the customization…
For those that you want to receive the email, leave
To field as the default, your customers' real email address:
For the other group, the “hold out” variant, set the
To field to your blackhole address (
blackhole@YOURDOMAIN.com). Do that by clicking the ‘Customize’ button in the
These emails will still be sent, but they will be sent to the blackhole address instead of your customers' real email address. Remember, ideally, this is an email address that you will ignore. Any interaction with the emails sent to your blackhole address (such as opens or clicks) will still be logged so it’s best if these emails can be discarded completely to avoid that.
When the test is set up, this is what it looks like:
In Customer.io, this will behave like a regular A/B test. You will be able to go to the A/B Tests tab in the campaign and see the data for both variants, including the conversion rates for the group which received no email.
Hopefully this helps you get up and running with blackhole/holdout tests – if there’s anything else we can help you with, please let us know. You can also check out our document on more complex cohort testing cases!