Setting up Blackhole/Holdout Testing

Introduction

A blackhole test (also known as a holdout test) is a special type of cohort testing: email vs. no email! You want to send some of your users an email, and others to receive nothing. This recipe will help you get that set up.

Note: If you'd like to test something other than content/no-content, such as timing or single email vs. multiple emails, we have a cohort testing recipe for you!

Ingredients

All you’ll need is:

  • Access to your email admin (in order to create a new email address)

Method

Step 1: Create a ‘blackhole’ email address

This is essentially an email address that you will ignore. Its purpose is to send some of your messages to an unmonitored address that you control instead of to your customers. This is usually named something like blackhole@YOURDOMAIN.com.

The email address you choose should be a real email address that you set up on a domain you control but that you DO NOT set up in an email client such as Outlook or Gmail. Setting up a real email address for your blackhole email address will prevent the messages that are sent to it from bouncing due to an invalid email address. Also, NOT setting up an email client to check the blackhole address will prevent our tracking pixels from getting loaded and causing your blackhole messages to be marked as opened by the email client even if they have never been opened.

If you plan to send a large number of messages to this blackhole address, you may wish to configure the blackhole email address (at the server level) so that these messages are automatically deleted in order to avoid exceeding any storage limits your email provider may have.

Step 2: Set up your A/B test

Within your campaign, add an A/B test to the email template you would like to test. This is done by clicking the email item to expand it and then clicking on the “Add A/B Test” button to the right of the item.

The ‘blackhole’ part comes next, in the customization…

Step 3: Customize the “To” fields

For those that you want to receive the email, leave To field as the default, your customers’ real email address:

For the other group, the “hold out” variant, set the To field to your blackhole address (blackhole@YOURDOMAIN.com). Do that by clicking the ‘Customize’ button in the To field:

These emails will still be sent, but they will be sent to the blackhole address instead of your customers’ real email address. Remember, ideally, this is an email address that you will ignore. Any interaction with the emails sent to your blackhole address (such as opens or clicks) will still be logged so it’s best if these emails can be discarded completely to avoid that.

When the test is set up, this is what it looks like:

In Customer.io, this will behave like a regular A/B test. You will be able to go to the A/B Tests tab in the campaign and see the data for both variants, including the conversion rates for the group which received no email.

Success!

Hopefully this helps you get up and running with blackhole/holdout tests – if there’s anything else we can help you with, please let us know. You can also check out our document on more complex cohort testing cases!