Scheduled syncs and real-time streams
There are two ways to get Salesforce data into your pipeline:
- Real-Time Stream: Stream data to your destinations in real-time.
- Scheduled Sync: Pull data from Salesforce into your pipeline on a regular interval.
Salesforce data) a-->b{Sync or Stream?} b--->|Stream|c(Send change to
Data Pipelines now) b-->|Sync|d((Wait until Sync
interval)) d-->e(Data Pipelines fetches
Salesforce changes)
In both cases, the result is the same: your Salesforce data goes through your pipeline. The only differences are:
- How up-to-date your destinations are with Salesforce data.
- How your Salesforce setup affects your Customer.io and Salesforce plans.
The option you choose may depend on your Salesforce plan. And, while Customer.io’s plans allow for a generous amount of API calls, we charge for incoming API calls—and syncs cost fewer API calls. You might need to balance your desire for up-to-date data against your bill!
Does my Salesforce plan support syncs or streams?
Before you go on, note that the option you pick may depend on your Salesforce plan. Each plan type has different API limits for real-time streams (Salesforce’s Apex feature) and for syncs (Salesforce’s Web APIs).
You might want to consult with your Salesforce administrator to determine your limits before you decide whether to stream or sync data to Customer.io.
| Salesforce Editions | Real-Time Stream (Apex) | Sync (Web API) |
|---|---|---|
| Enterprise | ||
| Professional | (requires Web API add-on) | |
| Developer | ||
| Unlimited | ||
| Database.com | ||
| Performance |
Real-time streams
When you set up a real-time stream, Salesforce pushes updates to Data Pipelines whenever your data changes. For example, imagine that you set up a real-time stream for Contacts. In this case, Salesforce sends data through Data Pipelines whenever you create a new contact, edit an existing contact, or delete a contact.
or delete data a->>b: Send data to pipeline b->>b: Convert to people,
events, objects b->>c: Send to destination
Scheduled syncs
Scheduled syncs tell Data Pipelines to pull data from Salesforce at a regular interval.
in Salesforce b->>a: Query for changes a->>b: Send changeset b->>b: Convert to people,
events, objects b->>c: Send to destination
While you can see incoming data in the Data In tab like most sources, syncs also show in the Imports tab. Here, you’ll see each scheduled sync operation and how many records were successful or failed. It gives you an at-a-glance understanding of the data that makes it into your pipeline.
Scheduled syncs use Salesforce’s API limits. These limits are fairly generous if you’re using Salesforce’s Enterprise edition or Professional edition with the API access add-on.
How Salesforce handles API limits
Salesforce has daily API limits which depend on your plan. These limits are shared across every application that uses the Salesforce API, not just Data Pipelines. When you reach an API limit, Salesforce stops sending data and responds with an error.
When you reach your limit, you’ll stop seeing data in Data Pipelines for the rest of the 24 hour period. If you notice a conspicuous lack of Salesforce data for a period of time, it’s likely that you’ve reached your Salesforce’s API limit(s). You can find errors related to API limits by inspecting your latest syncs in the Imports tab.
Syncs batch incoming data
Syncs batch incoming data up to 1000 records at a time. This means that a sync can process up to 1000 Salesforce records with a single Salesforce API call. This helps stay within your Salesforce REST API limits.
Real-time streams consume an individual Customer.io Data Pipelines call for each change in Salesforce. If you find yourself hitting Salesforce’s API limits with real-time streams, you might want to consider switching to syncs.
