In this article
One year ago, our small but mighty data team spent most of its time in the classic hamster wheel of a reporting-centric workflow. A question came in, we built a model in dbt, slapped a Sigma dashboard on top of it, and moved on to the next. Our team revisited content when stakeholders flagged issues, which pulled the team’s focus away from creating the new models and dashboards that would create the stakeholder confusion of tomorrow; it was time to change.
Over the last several months, we’ve gone all-in on a new way of operating. We’ve created what we call the ‘agentic middle’ of our Data team, helping us build and manage our data pipelines, create and test ML models, and transforming how we deliver self-service analytics via Smarty, our internal agentic data analytics platform and the focus of this post.
Reimagining self-serve analytics
We wanted to go beyond ‘cleaning up dashboards’ and create a new self-service experience, end-to-end. We knew we had some work to do on the data/semantic modeling side in Snowflake and dbt, so we documented core principles there and began building. It was also clear to us that natural language was the desired self-serve primary entry point for most stakeholders, so we started testing conversational analytics features within our existing stack, including Sigma's Ask Data and Hex Threads. We focused testing on clean and context-rich data sets and while results were promising, we found the answers thin, visualizations limiting, or the interface rigid and hard to scale.
Around this same time, OpenClaw, an open-source agent harness (the software around an AI model that gives it tools, memory and instructions so it can act), hit the scene and the agentic AI future everyone had been imagining seemed to arrive all at once. Many of us at Customer.io hopped on the hype train, having fun creating our own home agents, and thinking of use cases for agents at work. What we were trying to do with self-serve analytics seemed to be one of these use cases, and our CTO encouraged us to give it a shot.
We had the first prototype of ‘Smarty’ up and running within a week. First came the Slack agent that could take questions and answer back using our curated semantic views in Snowflake. These context-rich views carry definitions of each business metric, like ARR or churn, once, along with the dimensions you can slice it by, so every query result is built on the same definition. We also created a front-end site for Smarty to act as an internal BI-hub, admin control surface, and monitoring plane for usage and agent effectiveness. In those early days we developed Smarty through Smarty: the team and I worked with it in the OpenClaw console, or even in Slack, and the agent committed each change to its own repo. Today most of that work happens in Claude Code. The tool, and our means of managing it, have evolved since and Smarty is now the Data team’s main endorsed source for self-service analytics.
What Smarty does today
Since its inception, we've kept iterating on Smarty and expanding the platform underneath it. Smarty now runs on Talon, our internal agent harness, with Opus as its main model. Since the first real question in February, people have asked Smarty close to 2,000 questions, more than 300 of them in the last 30 days.
Here's how people use Smarty today:
- Conversations. Ask in Slack or on the Smarty site. Every conversation is saved, so you can come back to it, share it with a teammate, or start in Slack and pick it up on the site.
- Endorsed reports. The data team's trusted reports on core metrics like revenue, retention and pipeline. They're version controlled, and an "Ask a follow-up" button on each one opens a conversation with Smarty. There are ~40 live today.
- Smartifacts. When a conversation lands on something worth keeping, Smarty turns it into a Smartifact, a refreshable, shareable mini-report built around your question. You can ask Smarty to change it in place, each Smartifact gets its own short link, and the ones you're finished with retire to the Graveyard.
- Other agents. Agents across Customer.io can talk to Smarty directly to pull endorsed and authoritative KPIs.
Smarty is easy to use for both our stakeholders and the data team. Smarty creates all of its content via prompting, leveraging guides and templates we’ve built over time to ensure consistent patterns. Every chart is built in D3.js using our brand palette. Some stakeholders have even uploaded hand-drawn sketches of desired dashboards for Smarty to recreate. The agent does a great job on that first draft, and users can fine-tune it from there in conversation.
How Smarty uses semantic views to give consistent answers
The rub with any agentic analytics tool is getting deterministic answers from a non-deterministic system. If two people ask Smarty the same question, they need to get the same number.
So we drew a hard line: Smarty answers only from our governed semantic views (SVs). It can't explore Snowflake or write its own SQL against other tables. When a question isn't covered, Smarty says the data isn't available yet and offers the closest proxy it can find, clearly labelled as a proxy.
We considered other approaches. A common one is a tiered fallback: use a governed metric when one exists, and let the agent write its own SQL against trusted tables when it doesn't. Some companies go further and trust frontier models to reason their way to accurate metrics from the warehouse and its documentation. We're continuing to test that, but we've seen more consistent results from SVs. Instead of giving Smarty broader Snowflake access, we hold the line at SVs and continue to expand coverage over time.
When Smarty can't answer something, that gap becomes work for the data team. Today Smarty reads from 14 SVs spanning revenue, retention, pipeline, billing, product usage, messaging and NPS, up from 6 in April, and they cover most of what the business asks. We continue to support teams and agents that need access to raw data, while positioning Smarty as our endorsed, authoritative source of truth.
We also manage .md guides in addition to the context provided to the agent via the SVs. Our analytics guidelines hold broadly applicable rules, like when a month counts as complete and how to handle percentiles properly. This is also how Smarty knows to always answer objectively and avoid charged language, etc. We manage business context guides for different areas of the business, giving Smarty more company and function-specific context. The context guides and SV info are also used by Smarty to maintain a KPI documentation page on the site to ensure stakeholders have clarity on metric definitions and nuance used by the agent.
Governance and access control for AI analytics agents
Because Smarty is the endorsed path to our data, it gets the same access controls as any production system. There is role-based access control (RBAC) within Smarty itself, and user permissions can be managed at the Smarty-global (view, create, admin) and content-specific level. Smarty's backend validates user permissions on the site or via Slack, and users have a simple way to request new or elevated access.
The Smarty ‘Observatory’ allows the admins to watch Smarty at work. It shows us what people are asking about, through what channels, and where its blind spots are based on its derived confidence score when it answers a question. ‘Hot Spots’ are grouped themes Smarty identifies as needing human attention, which are raised to the Data team for review to consider expanding context or SV coverage. A daily scan keeps all of it current.
What's next for Smarty
The frontier keeps moving, so we're constantly evaluating where we can improve the experience for our users and the effectiveness of the agent. Right now, we're focused on these areas.
- Automated Context Improvement. We are working to go beyond the manual review of Observatory ‘hot spots’ and use our broader network of agents to keep Smarty’s context current, and automatically raise PRs for plugging gaps.
- Full coverage. There are some remaining gaps in Semantic View coverage. Once these gaps are closed, we can further converge on Smarty as our single source of truth.
- More agents using Smarty. We have literally dozens of agents at Customer.io helping with all sorts of tasks throughout the company. Many of these agents have direct access to raw data and when asked for KPIs, can derive a number that isn’t aligned with our SVs. We’d like these agents to ask Smarty these questions as we expand our SV coverage over time.
How to build your own agentic analytics platform
A few things we'd tell any team starting out:
- Pick a point of governance and stand behind it. Define your most-asked metrics once in semantic views, and have your endorsed agent answer only from them. Grow that coverage over time, and keep other routes to raw data open for the people who need them.
- Keep humans in charge of the context. Agents can draft updates to your guides, but a person on the team reviews and approves every change. A short guide per business area, owned by the people who know it, does more than any amount of raw schema.
- Ask for facts. Tell the agent to report numbers and leave the judgment to people. Trust builds faster that way.
- Watch what people ask. Log every question, score confidence and turn the gaps into tickets. That backlog becomes your roadmap.
- Build it with the agent. We developed Smarty by talking to it, and the speed surprised us.
If building tools like Smarty sounds like your kind of work, check out our open roles and apply today.









