A/B testing campaign messages
A/B testing lets you test different versions of email, push notifications, and SMS messages in your campaign, so you can measure which version performs better. For email, this means you can test subject lines against each other, different versions of your call to action, or try out new designs. For SMS and push notifications, you can test for content, imagery, or even deep links.
You can test multiple messages in every triggered campaign, though we do recommend sticking to making just one change in your variation. This allows you to get an accurate picture of how that change measures up against the original without being confused by other data points.
How do I set up an A/B test?
For either email, Push, or SMS messages, this process is the same. First things first, you’ll want to head to the campaign where you want to run your test, and create the type of message you want to test—email or push notification.
Then, on the message you’d like to add a test to, you’ll see a Turn into A/B Test button.
Want to run a hold-out test instead?
Use a Random Cohort Branch to run a test in which some members of your audience receive a message and the rest don’t. Learn how to run a hold out test.
Clicking this creates an A/B test workflow item. Here’s an example with SMS:
By default, 100% of the traffic goes to the original and 0% to the variation.
Make your changes to the variation. You can change most anything about the message. In the case of email, this could be your from address, subject, body, even the sending mode (“Queue Draft” vs. “Send Automatically”). For push, feel free to test your push title, body content, or get creative with custom payloads. Any delays and time windows are shared across both versions, though.
We start routing traffic as soon as your campaign is live.
If you have a low volume of messages being sent or are just starting with A/B testing, try testing a small subject line or content change, and measuring opens.
Hold out testing for emails
Rather than testing how your audience responds to variations of a message, a holdout test prevents some members of your audience from receiving a message at all—they’re purposely “held out” of the test. This helps you determine whether a message is useful or not. See Holdout Testing for more information.
Either variation in your A/B test—email A or B—can be the holdout. But, for this example, we’re treating variation B as the holdout.
In your campaign’s Workflow, select the email you want to convert to a holdout test.
Click Turn into A/B test.
Set the amount of traffic that you want to send to your message (variation A) and holdout group (variation B).
Click Edit Variation B, and select Don’t deliver this message to people. This converts the “message” to a holdout, ensuring that it’s sent internally to a message trap; it is not sent or visible to your audience.
Now, when you start your campaign, some people will receive Message A, and some people won’t get a message at all.
Check in on your changes
Once you have an A/B test running in an active campaign, you should see an A/B Test tab appearing in your campaign overview. That’s where you can go to see the results of the campaign and pick a winner.
When you have statistically significant results, or if you want to end the test before that, you can pick one of the options as the winner by clicking the “Select winner” button. We’ll remove the other option and remove that particular A/B test from the screen.
After you end the test, you will no longer be able to see your non-winning content or your A/B test results.
Understanding your test results
Want more information about what statistical significance means or how we’re calculating different things? Read about it in more detail here.
