Webinar

Agent-recommended experiments: Q4 2026 launch webinar recap 

Runtime:
00:57:50
Air date:
Speakers:
Molly Evola
Molly Evola
Sr. Content Marketing Manager

The week before Black Friday has a familiar routine. Someone grabs last year's email, tweaks the subject line, runs one quick A/B test, and hits send. A single copy test in the last few days before the sale rarely shows enough about conversion to build on.

Our Q4 2026 launch webinar, Smarter marketing through agent-recommended experiments, walked through a different route. Our product team used a fictional online learning platform called Skillwise to show it. The Agent analyzed last year's Black Friday data and then drafted on-brand emails in Design Studio, and an experiment with a holdout tested the Agent's recommendation against the original email. Each step shaped the next one, so the data picked the test and the test informed the copy.

Ed Preston, who leads CRM at Cleo, joined to show how Cleo uses these features for email builds, a Spanish-language launch, and faster testing. Andrea from our product marketing team then covered the rest of what shipped, from sign-up forms to a certification exam.

TLDR

  • Customer.io's Agent can now answer questions about your workspace data, including message performance, site and app activity, profiles, attributes, and objects, either inside the platform or through an MCP client such as Claude.
  • Experiments test whole paths within an automation, such as email only versus email plus push, with a holdout that measures lift over doing nothing and automatic winner selection as soon as a result is significant.
  • Design Studio is the default email editor, and the Agent reads your global styles, brand document, and reusable components before it drafts, so first versions already match your brand.
  • Cleo builds every new email in Design Studio, uses language rules to localize emails for its Spanish launch without an external translation agency, and lets experiments publish winners automatically.
  • Seven more updates shipped, from sign-up forms to a free certification exam, and agentic experimentation reaches everyone in Q1 2027.

Agent analytics: turn last year's Black Friday data into this year's plan

Agent analytics lets you ask questions about your workspace data in plain language and get charts and context back in seconds, with no dashboard slicing or CSV exports. Scott opened the Agent inside Customer.io and started with a broad question about how Black Friday affected our fictional company, Skillwise, purchases last year.

Behind the scenes, the Agent has access to your message performance data, your website and app activity, your profiles and how they're growing, and your attributes and objects. Because Customer.io holds event data alongside delivery data, the Agent can answer questions about what customers did after a message landed. Skillwise stores its courses as custom objects, so the Agent could break purchases down by course too.

Design Studio: an on-brand first draft, with the QA built in

Design Studio is now the default email editor in Customer.io, and the Agent drafts in it after studying your brand. Carmen's demo picked up once the Agent had finished a Black Friday email for Skillwise. Before it created anything, the Agent read the workspace's global styles and brand document, looked at existing email designs to learn Skillwise's formatting patterns, checked course data to decide which courses to feature, and reviewed the custom components already built, including the header, course cards, and footer.

The result looked familiar to the Skillwise team. It used the brand colors, fonts, and button styles, and it chose Skillwise's sale promotional button style for the final button because the global styles spelled out when each style applies. The email featured three courses, while Scott's analysis had recommended leading with plans, which made it a natural counterpart for the experiment later in the demo.

Global styles, the brand document, and components

Global styles hold the colors, heading formats, and button styles shared across the workspace, so every message stays consistent. The brand document, which the Agent generates from your product, sits at the bottom of that page. The Agent reads both before it creates a message, so the styling and tone match your business. Components are shared across every email that uses them, so a new privacy policy link in a footer component takes one edit that applies everywhere. Carmen recommended setting up global styles if you haven't yet.

Built for designers and developers

Design Studio shows the email in code as well as in the visual editor, so nobody passes HTML back and forth or requests style updates in the code.

Check it before you send

Design Studio keeps testing and QA in the editor, so you don't need to leave the platform to review an email.

Design Studio also supports notification inbox, SMS, and WhatsApp, with more channels coming soon. In-app is the next channel in testing, and SMS needs your own SMS provider account connected in workspace settings for now.

Our take: Global styles are the setup step worth doing first. They make the Agent's first draft look familiar to your team, so your review time goes to voice and judgment.

Experiments: test whole paths, then let the winner promote itself

Experiments in Customer.io now run across branches within an automation, so you can test whole paths, such as email only versus email plus push, different sequences, or different timings, and add a holdout to measure true lift over doing nothing. Zach noted that experiments replace the earlier A/B test functionality in automations, and every result stays on record so you can build on what worked.

He set up Skillwise's test two ways, by asking the Agent and by clicking through the setup himself. The automation had a single trigger, a group of people, and a conversion event for purchases.

Setting up the experiment with the Agent

Zach's prompt:

Turn the email in this automation into an A/B experiment. Keep the current email as variant A and draft a new email as variant B using our styles and assets, with a different focus: getting people to buy an annual or monthly plan, and highlighting the free AI course. Split the audience between the variants, measure success with the same conversion event as the automation, and pick the winner automatically.

The Agent came back with a description, the automation's conversion event as the success metric, two variants (the current email and a plan-focused one), a holdout, automatic winner selection, and a rule to pick the variant with the best rate if the test ends inconclusive. It also rewrote the second email around plans, so the test was ready to run.

Setting up the experiment by hand

The manual setup covers the same ground, in this order:

  1. Add a description of what you're testing.
  2. Choose the success metric. An engagement metric comes from a recipient performing an event, and a conversion event comes from your automation. A button copies the metric from the automation, and you can add data filters below it.
  3. Name the variants. Skillwise used "course focus" and "plan focus."
  4. Add a holdout.
  5. Pick how the winner is selected: manually, automatically at the end of the experiment, or automatically as soon as a variant is significant. With Black Friday close, Skillwise chose as soon as significant, and everyone else then receives the winning message.
  6. Decide what happens if the experiment is inconclusive. Skillwise picked the variant with the best rate.
  7. Set how long the experiment runs. Skillwise chose 14 days.

Reading results and applying a winner

Once the automation is live, the experiment shows a running total of the people who have reached the test, plus results, paths, and settings. In one example from the demo, variant B pulled clearly ahead and the experiment flagged the result as significant, so there was no need to wait for the full run. Choosing a winner opens a summary the Agent has filled in about what mattered in the test, and you can add your own context. Selecting the winner routes everyone else through it. Applying the winner then removes the experiment from the automation, leaving a beaker icon as a breadcrumb and a record in the experiments tab, which lists every experiment you've run in that automation.

Experiments also work with language variants, so you can test translated versions. To dig into who saw what, open the people tab of an experiment, click through to a variant, and build a segment from everyone who went through it. Each person's activity shows their experiment data too.

Zach closed the demo with the email sending. After about two weeks, the Skillwise team can check whether plans beat a single course, which shows whether the Agent's insight held up.

Our take: Before you build any experiment, settle the one question the test should answer, the metric that reflects the outcome you care about, and whether to add a holdout. Then bring in the Agent for suggestions and choose which ones to run, as outlined in our release overview.

How Cleo uses Design Studio, language rules, and experiments

Cleo, an AI money app with a personality, uses Design Studio, language rules, and automatic winner selection to build emails faster, launch in Spanish, and run more tests. Ed Preston has led CRM at Cleo for more than two years and has spent over 15 years in CRM and email marketing. Cleo is used by more than four million people, and it's available around the clock to tell them the truth about their spending, even when it stings. Its mission is to change the world's relationship with money.

From pasted HTML to Design Studio

Cleo used to build emails in a basic drag-and-drop external tool and copy the raw HTML into Customer.io. That worked at a smaller scale with one region and one language, but it fell short on consistency. Since the start of this year, every new Cleo email has been built directly in Design Studio. Shared components keep the look and feel consistent, and one edit to a shared component, even something as small as padding, carries across every email that uses it. The old tool also produced oversized images and hero images that cut people's heads in half. Those problems are gone, and dark mode is handled.

Roughly 30% of Cleo's emails are in Design Studio today, and the other 70% remain in legacy HTML. Ed expected to reach 100% coverage by the end of the following week, with the Design Studio team helping automate the move, which Cleo's setup of templates and components makes possible.

Review panel and image optimization

Ed aims for 100% green in the review panel before anything sends, and most emails get there now. The review panel used to report a missing unsubscribe link when the link sat in a snippet inside a component, and Customer.io has updated the check so it recognizes those links.

Image size is the other recurring item, and Ed demoed it live on a Black Friday email. The email scored 76%, with links and Liquid passing and an unsubscribe link in place. The only open issue was two images, a desktop version and a mobile version, sitting just above Cleo's 100 KB target. Optimization comes in several levels. The maximum setting cut 41% from the first image and moved the score to 88%, and a regular optimization on the second image brought it to 100%, with no difference visible to the naked eye.

Spanish translations in Cleo's voice

Cleo operates in the US and the UK, both in English, and handles the small spelling differences itself. Next month, Cleo launches a Spanish version of its US app, so its emails need Spanish too. Cleo trialed Customer.io's native translation from US English into Spanish, and Ed scored the out-of-the-box result six out of ten. His Spanish-speaking content designers found it fine, but it missed Cleo's distinct tone of voice, and the other option was sending the work to an external agency.

Cleo shared examples of the changes it wanted with the Customer.io team, and language rules grew out of that feedback. Cleo added its copy guidelines plus about 30 words and phrases to avoid, each paired with what to use instead, and the list keeps growing. In one example, the first translation used the masculine form of the word for member. The updated version avoids it and speaks to customers in a more relaxed, casual way. After a couple of iterations, Cleo no longer needs the agency, and keeping translation in the platform saves the loop of exporting copy and importing it back.

Experiments that finish themselves

Before this release, Cleo set a test live on a Monday morning and checked back a week later for a winner. The winner might have emerged two, three, or four days earlier, and when there was no winner, the team waited another week. Now the winning message goes live automatically as soon as the test reaches statistical significance or whichever threshold Cleo sets. Ed expects that to raise the number of tests and variants Cleo runs at once.

Ed showed two live tests on branches of one campaign:

  • A design test: a simple A/B test with no holdout and the same subject lines, comparing a new email design with the current one.
  • A subject line test: four variants at 25% each, measured on clicks, with the same design throughout. It compares "Let's get you set up" and "Welcome to Cleo" with and without the recipient's first name.

Questions from the chat, answered

Here are the attendee questions we answered live, with the answers as given during the session.

Will new channels include RCS? Yes, RCS is planned for Q1 2027.

Can the Agent look at event attributes? Yes. You can use events and their attributes when you ask the Agent questions.

Can I turn the Agent's answers into a report? Charts and reports live in the Agent chat for now. The team is exploring ways to make them more accessible and shareable.

Can I use my existing Claude design system to set global styles in Design Studio? Yes. When you run brand generation, you can include images, docs, code, and email files, and you can add your existing Claude design system at that point.

Are in-app and SMS available in Design Studio? In-app isn't live yet. It's the next channel and it's in testing now. SMS shows up in Design Studio after you connect SMS in your workspace settings, and for now it works only with your own SMS provider account. SMS purchased through Customer.io isn't supported yet, and the team is working on it.

Can I apply multiple brand styles in one workspace? Not at the moment. Global styles are per workspace, so a workspace has one brand or theme.

Can two people edit the same template at the same time? Not today. If someone opens an email a teammate already has open, a pop-up shows who's editing and lets them take over or view only. If simultaneous editing matters to your team, email our product team to help shape the roadmap.

How do the Agent and MCP prevent an automated agent from sending live messages to the wrong audience? The Agent has two modes, ask and auto. In ask mode, it requests approval before making any live change. In auto mode, a small classifier responds to the approval request for you. For MCP, live edits are a toggle you can enable or disable at the workspace level, and your MCP client, such as Claude or ChatGPT, then has to approve changes in its own layer.

Where to start in your own workspace

Start by asking your Agent what to test in your highest-volume automation. These updates are available on all Customer.io plans Essentials and up, so you can try the Skillwise sequence this week:

  1. Ask the Agent a diagnostic question about your own data, such as which automation has the most volume but the lowest conversion rate, using the prompts in our release overview as a starting point.
  2. Set up global styles in Design Studio if you haven't, so the Agent's first drafts look familiar to your team.
  3. Build an experiment with a holdout, with a clear question and a success metric chosen before you launch.
  4. Take the free certified practitioner exam in Academy when you want credit for what you know.

To go deeper, read about what the Agent can do or book a demo for a walkthrough of everything Customer.io can do for your messaging program.

Supercharge your messaging 

  • 14-day free trial
  • No card required
  • Cancel anytime