How to A/B test an entire customer journey 

To A/B test an entire customer journey, randomly assign people to a version of the journey when they enter it, give each group a different sequence, and compare everyone on one goal, such as activation or purchase, over a fixed window. Add a holdout group that receives no journey so you can see the lift you created.

A subject line test tells you which email gets opened more. Questions about sequence, timing, and channel mix get answered when you test the journey as a whole: whether a five-email onboarding flow beats a three-email one, or whether a push notification on day two helps people activate.

The metric you care about points the same way. According to our Customer messaging in 2026 guide, 66% of marketers track conversion rate as their primary success metric, and 34% track open rate. Conversion happens at the end of a journey, so a journey-level test lines up with the number you report.

This guide covers what to vary, how to size and structure the test, a real customer example, and how to build it in Customer.io.

Journey-level A/B testing compares two or more complete versions of a multi-step customer journey, with each person experiencing only one version from entry to goal. The winner is the version that gets more people to the outcome you care about, such as a first purchase, a completed setup, or a renewal.

Messages in a journey influence each other. A strong welcome email changes how someone responds to the follow-up three days later, and an extra reminder can win on clicks while costing you subscribers by the end of the week. A single-message test can't see those interactions because it stops at the edge of one send.

Both tests have a place in a lifecycle program. Message tests polish individual sends, and journey tests decide the structure those sends sit inside.

Journey tests vary the structure of the experience, so the best candidates are the decisions you made once and never revisited. Seven are worth putting on your test backlog.

Our docs show a good starter case: a free-to-paid conversion campaign that uses a random cohort branch to test the best time to send an offer. Timing questions like that one are cheap to set up and often produce a clear result.

Pick the variable your team argues about most. A decision that has stayed unresolved in meetings for a quarter could be a decision with real revenue attached.

A journey A/B test takes seven steps: choose a journey and goal, write a hypothesis, change one structural variable, size the test, randomize at entry, add a holdout, and read results after a fixed window.

1. Choose one journey and one goal. Pick a journey with steady volume, such as onboarding, abandoned cart, or win-back. Define the goal as a specific event within a specific window, for example "created first campaign within 14 days."
2. Write a hypothesis that names the structure. A useful one reads like this: adding a push notification on day two will raise 14-day activation by 10% relative to the email-only journey, with no increase in unsubscribes.
3. Change a single structural variable. Keep copy, design, and offer identical across versions wherever you can, so any difference in results traces back to the variable you changed.
4. Size the test before you launch. With a 10% baseline conversion rate, detecting a lift to 12% at 95% confidence and 80% power takes roughly 3,800 people per version. Smaller lifts need more people, so check your entry volume against the time you're willing to wait.
5. Randomize when people enter the journey. Assignment happens once, at entry, so each person sees one complete version. In recurring campaigns, store the assignment as a profile attribute so people stay in the same group on later entries.
6. Add a holdout group. A common starting point is 5% to 10% of entrants who receive no journey. The holdout tells you how much of your result comes from the journey itself.
7. Run the test for a fixed window, then read the goal metric. Set the end date from the journey length plus your conversion window, and check guardrail metrics such as unsubscribes and spam complaints before you declare a winner.

Monarch Money used holdout testing to validate a rebuilt trial onboarding journey, and the new journey delivered a 3.36% drop in trial cancellations, a 4.4% lift in reports page views, and a 2.5% lift in users updating categories.

The original journey was a time-based sequence of four generic emails sent over the course of a trial. Sue Cho, Monarch's Head of Lifecycle Marketing, saw slightly higher cancellations and no incremental lift in engagement from that flow. Her team mapped the actions that matter during a trial, such as connecting a bank account or updating transaction categories, and built triggered emails that send a single, timely nudge based on in-app behavior. Product usage data reaches Customer.io through a Segment integration.

In Customer.io, you test a full journey by placing a random cohort branch at the start of a campaign workflow and building each version of the journey on its own path. A random cohort branch supports up to 20 paths, and the percentages across paths must add up to 100%.

Here's how the setup works:

1. Add a random cohort branch right after your campaign trigger, so assignment happens at entry.
2. Build version A on the first path and version B on the second, including the delays, channels, and branches you want to compare.
3. Add a third path with no messages if you want a holdout, and give it 5% to 10% of the split.
4. Set the campaign goal to the conversion event you chose in your hypothesis, so each path reports against the same outcome.
5. Launch, and compare goal conversion across paths once your planned window closes.

A few details from our docs help you avoid surprises:

• When you drag a random cohort branch into an active campaign, one path starts at 100% and the others at 0%. That gives you time to build the new paths before anyone enters them, and you raise the percentages once the build is done.
• Random distribution needs volume to settle. Small samples may not match your configured percentages exactly.
• If you use custom objects, the "Cohort by" setting can group people by company or account, so everyone in the same account follows the same path. People who belong to more than one object of the same type exit the campaign immediately, so choose "People" when that applies to your data.
• In recurring campaigns, the split is evaluated on each entry, so someone can land on a different path next time. Our support team suggests tagging people with an attribute that records their group when you need consistency.

For tests that compare a single message against no message, holdout tests let you turn a message into an A/B test and hold one version out entirely. Use the random cohort approach when the thing you're testing is the whole sequence.

Compare the goal rate of each group at the end of your fixed window, then check that the gap is larger than random noise and that guardrail metrics held steady. Most journey tests that go wrong fail at one of these points:

• Calling a winner early. Dashboards show a leader within days. Checking repeatedly and stopping at the first lead inflates false positives, so commit to your end date before launch.
• Reading opens and clicks as the result. Those numbers describe attention on individual messages. The journey's job is the goal event, so rank versions by goal conversion, revenue per person, or retention.
• Changing percentages mid-test. People who entered under different splits have had different exposure time, which muddies the comparison. Set the split at launch and leave it.
• Testing several things at once. If version B has a new channel, new timing, and new copy, a win teaches you nothing about which change mattered.
• Ignoring other campaigns. A person in your onboarding test may also be receiving a newsletter and a promotion. Random assignment spreads that noise evenly, and pausing competing sends for the test window shrinks it.
• Skipping the log. Record the hypothesis, dates, sample sizes, and result for every test. After six months, the log shows which levers move your goal in your audience.

When a test ends without a clear winner, you've learned that the variable doesn't move your goal enough to matter at your volume. That frees you to spend effort on a different lever.

Yes. You randomly assign people to different versions of the full journey at entry, let each person move through one version, and compare how many reach the same goal. Marketing automation platforms with branching, including Customer.io, support this with random split or cohort branches.

An A/B test compares two versions of something people receive. A holdout test compares a version against a group that receives nothing, which measures the lift your messaging creates. Many teams run both: A/B tests to pick the best journey, and a holdout to confirm the winner beats silence.

Run it for the full length of the journey plus your conversion window, and at least until you reach the planned sample size. A seven-day onboarding journey with a 14-day goal needs about three weeks after the last person enters, and a cycle that covers a full week of traffic patterns.

The number depends on your baseline conversion rate and the lift you want to detect. As a reference point, detecting a lift from 10% to 12% at 95% confidence and 80% power takes roughly 3,800 people per version. Smaller lifts need more.

Measure one primary goal, such as activation, purchase, or retention at 30 days, and track guardrails alongside it: unsubscribe rate, spam complaints, and support contacts. Opens and clicks work as diagnostics for individual messages.

Yes. Add a random cohort branch to a campaign workflow, build each journey version on its own path, and compare conversion across paths. A branch supports up to 20 paths, and you can add a path with no messages to serve as a holdout.

The next journey test worth running is probably sitting in a meeting agenda right now: how long to wait between messages, whether to add a second channel, or how many follow-ups are too many. Write it as a hypothesis, set up a random cohort branch, and let the numbers settle it. Our docs on random cohorts walk through the build.