Scheduled syncs and real-time streams

There are two ways to get Salesforce data into your pipeline:

  • Real-Time Stream: Stream data to your destinations in real-time.
  • Scheduled Sync: Pull data from Salesforce into your pipeline on a regular interval.
flowchart LR a(Create, update, or delete
Salesforce data) a-->b{Sync or Stream?} b--->|Stream|c(Send change to
Data Pipelines now) b-->|Sync|d((Wait until Sync
interval)) d-->e(Data Pipelines fetches
Salesforce changes)

In both cases, the result is the same: your Salesforce data goes through your pipeline. The only differences are:

  • How up-to-date your destinations are with Salesforce data.
  • How your Salesforce setup affects your Customer.io and Salesforce plans.

The option you choose may depend on your Salesforce plan. And, while Customer.io’s plans allow for a generous amount of API calls, we charge for incoming API calls—and syncs cost fewer API calls. You might need to balance your desire for up-to-date data against your bill!

Does my Salesforce plan support syncs or streams?

Before you go on, note that the option you pick may depend on your Salesforce plan. Each plan type has different API limits for real-time streams (Salesforce’s Apex feature) and for syncs (Salesforce’s Web APIs).

You might want to consult with your Salesforce administrator to determine your limits before you decide whether to stream or sync data to Customer.io.

Salesforce EditionsReal-Time Stream (Apex)Sync (Web API)
Enterprise
Professional(requires Web API add-on)
Developer
Unlimited
Database.com
Performance

Real-time streams

When you set up a real-time stream, Salesforce pushes updates to Data Pipelines whenever your data changes. For example, imagine that you set up a real-time stream for Contacts. In this case, Salesforce sends data through Data Pipelines whenever you create a new contact, edit an existing contact, or delete a contact.

sequenceDiagram participant a as Salesforce participant b as Data Pipelines participant c as Destinations note over a, c: For each create, update, or delete operation a->>a: Create, update,
or delete data a->>b: Send data to pipeline b->>b: Convert to people,
events, objects b->>c: Send to destination

Scheduled syncs

Scheduled syncs tell Data Pipelines to pull data from Salesforce at a regular interval.

sequenceDiagram participant a as Salesforce participant b as Data Pipelines participant c as Destinations note over a, c: Sync Data: occurs on a schedule a->>a:Changes happen
in Salesforce b->>a: Query for changes a->>b: Send changeset b->>b: Convert to people,
events, objects b->>c: Send to destination

While you can see incoming data in the Data In tab like most sources, syncs also show in the Imports tab. Here, you’ll see each scheduled sync operation and how many records were successful or failed. It gives you an at-a-glance understanding of the data that makes it into your pipeline.

We show a sync of contacts (people) that completed in the Imports tab
We show a sync of contacts (people) that completed in the Imports tab

Scheduled syncs use Salesforce’s API limits. These limits are fairly generous if you’re using Salesforce’s Enterprise edition or Professional edition with the API access add-on.

How Salesforce handles API limits

Salesforce has daily API limits which depend on your plan. These limits are shared across every application that uses the Salesforce API, not just Data Pipelines. When you reach an API limit, Salesforce stops sending data and responds with an error.

When you reach your limit, you’ll stop seeing data in Data Pipelines for the rest of the 24 hour period. If you notice a conspicuous lack of Salesforce data for a period of time, it’s likely that you’ve reached your Salesforce’s API limit(s). You can find errors related to API limits by inspecting your latest syncs in the Imports tab.

Syncs batch incoming data

Syncs batch incoming data up to 1000 records at a time. This means that a sync can process up to 1000 Salesforce records with a single Salesforce API call. This helps stay within your Salesforce REST API limits.

Real-time streams consume an individual Customer.io Data Pipelines call for each change in Salesforce. If you find yourself hitting Salesforce’s API limits with real-time streams, you might want to consider switching to syncs.

Copied to clipboard!
  Contents
Is this page helpful?