A/B Testing Newsletters

Testing messages is a great way to tune your newsletter for maximum impact. Adding a test allows you to evaluate any aspect of your email (subject line, content, design, you name it!) with variations, and send it to a sample percentage of people. Once sent, you can track data about the performance of each variation and send the most successful to your remaining recipients.

To test your newsletter, click the Add Test button on the “Content” step of your newsletter creation workflow:

From here, you can send your tests to a sample percentage, and automate sending to the rest. The button to add variations also appears on the left, below your original email. Adding a variation will copy your first message, so tweaking one variable of the email should be easy!

Sample Percentage

Let’s say you want 20% of recipients to receive one of two emails with different subject lines. A randomly-selected 10% of people will receive each email, and you can choose to automatically send the one which performs better to the remaining 80% once the test is over. Start by setting the Sample percentage to 20% using the slider:

Variations

To create your second email, click the Add variation button. You can test just about anything with your new variation: subject, from address, or the main content. However, we highly recommend only changing one variable per test to ensure the test is reliable. In our example, we’ve edited the subject line.

Note: you can add up to 8 variations for finely tuned testing, but keep in mind that the results are most useful with at least 500 recipients per variation. For fewer recipients, samples are not statistically significant. Learn more about Getting Conclusive Results from A/B Tests.

Choose Winner Automatically

If you would like for us to send the winning message to the remaining recipient list for you, toggle the Automatically send a winner to remaining recipients? option to Yes. Select the metric you’d like us to use to decide the winner and how long the test should run before sending it to the remaining recipients.

Using the subject line example from above, testing the open rate will tell you which subject line drew the most people in.

Note: Winner selection will not take the statistical significance of the test into consideration. To calculate the confidence level of the test yourself, please see Getting Conclusive Results from A/B Tests.

If you would prefer to analyze the results yourself and send to the remaining recipients manually, keep the Automatically send a winner to remaining recipients? toggled Off. Once the original newsletter has sent, you will have the option to send to the remaining recipients from the Tests tab of the newsletter overview.

Follow-ups and Changes

As soon as the newsletter sends, you can start to monitor the results of your test on the Test tab of the newsletters overview page.

From here, you can view the current metrics for each variation, with the metric chosen to track the winner in bold fonts. The Details section of the page is where you can see the timeline of the test. It tells you when the original newsletter was sent and to what percentage of your recipients, how much longer the test will run, and when the winning message will be sent to the remaining recipients.

Modify automation settings
If you would like to cancel the automation, edit the duration of the test, or edit the metric used to determine the winner, click the Edit link within Details:

Make any changes necessary in the modal, and click Save. Note that the duration of the test cannot be set to a value shorter than the amount of time that the test has already been running.

Cancel automatic send
If you no longer want the winner to send automatically, click the Edit link in the bottom-most timeline entry of the Details section. Toggle the Automatically send a winner to remaining recipients? toggle control to Off and save your changes. Once the automation has been turned off, it cannot be turned back on. You will always have the chance to send to the remaining recipients manually, but ensure this is what you want before saving your changes.

Send to remaining recipients manually

If you did not choose to automate sending your message to the winner, you will still see the metrics for each variation reported on the page, and the option to send one of your variations to the remaining recipients. Once you have determined that your test results are sound (see Getting Conclusive Results from A/B Tests for help in determining whether the results are conclusive or not), click the Send to remaining button. (Hint: you can also choose to Send to remaining directly from the variation row in the results table). In the modal, select the variation you would like to send and when you would like to send it. You can choose to send the variation immediately, schedule it to be sent in the future, or to not send it at all.

Completed test

When a test has been completed, all the metrics for the test will remain visible on the Test page, as well as all the details and timeline of the completed test.