A/B testing campaign messages
A/B testing allows you to test various aspects of both email and push messages in a campaign, and measure if the change you've made will perform better than the original. For email, this means you can test subject lines against one another, your call to action copy, or try out a new design. For push notifications, this means that you can test for content, imagery, or even deep linking.
You can test multiple messages in every triggered campaign, though we do recommend sticking to making just one change in your variation. This allows you to get an accurate picture of how that change measures up against the original without being confused by other data points.
How do I set up an A/B test?
For either email, Push, SMS, or Urban Airship Push messages, this process is the same. First things first, you'll want to head to the campaign where you want to run your test, and create the type of message you want to test-- email or push notification.
Then, on the message you'd like to add a test to, you'll see an Add A/B Test Button.
Clicking this creates an A/B test workflow item. Here's an example with Push:
By default, the A/B test is inactive as indicated by the small "Inactive" label, so 100% of the traffic goes to the original, and 0% to the variation. Leave this as-is while you make your change(s).
Make your changes to the variation. You can change most anything about the message. In the case of email, this could be your from address, subject, body, even the sending mode ("Queue Draft" vs. "Send Automatically"). For push, feel free to test your push title, body content, or get creative with custom payloads. Any delays and time windows are shared across both versions, though.
If you have a low volume of messages being sent or are just starting with A/B testing, try testing a small subject line or content change, and measuring opens.
When your changes are ready, give your message a 50 / 50 split (or 80/20, 30/70...it's up to you) by clicking the A/B test bar.
Check in on your changes
Once you have an A/B test running in an active campaign, you should see an A/B Test tab appearing in your campaign overview. That's where you can go to see the results of the campaign and pick a winner.
When you have statistically significant results, or if you want to end the test before that, you can pick one of the options as the winner by clicking the "Select winner" button. We'll remove the other option and remove that particular A/B test from the screen.
Understanding your test results
Want more information about what statistical significance means or how we're calculating different things? Read about it in more detail here.