10

Amazon Redshift Reverse ETL Source

Import data from Amazon Redshift to any destination. This source saves you the trouble of writing code to extract, transform, and load data from your warehouse. Instead, specify the query (or queries) to run, and we’ll handle the rest.

Best Practices

Before you add a Reverse ETL source, you should take some measures to ensure the security of your customers’ data and limit performance impacts to your database and Customer.io workspace.

• Create a new database user/service account. Implement a database user with minimal privileges specifically for Customer.io import/sync operations. This person only requires read permissions with access limited to the tables you want to sync from.

• Avoid using your main database instance. Consider creating a read-only database instance with replication in place, lightening the load and preventing data loss on your main instance.

• Sync only the data that you’ll use in Customer.io. Limiting your query can improve performance, and minimizes the potential to expose sensitive data. Select only the columns you care about, and make sure you use the {{last_sync_time}} to limit your query to data that changed since the previous sync.

• Limit your sync frequency so you don’t sync more than necessary and consume unnecessary resources. If the previous reverse ETL operation is still in progress when the next interval occurs, we’ll skip the operation and catch up your data on the next interval. You should monitor your first few reverse ETL intervals to ensure that your sync doesn’t impact your system’s security and performance—frequently skipped operations may indicate that you’re syncing too often.

Granting us access to your database

We support both SSL and non-SSL database connections. As a part of setup, you’ll need to provide the credentials of a database user with read-access to the tables you want to select data from.

If you use a firewall or an allowlist, you must allow the following IP address so that we can connect to your database.

Set up a Service Account

When you set up an Amazon Redshift source, you should create a user account that Customer.io will use to connect to Redshift. This user has limited access to your Redshift instance and gives you granular control over access to your Redshift instance.

1. Log in to Redshift and select the Redshift cluster you want to use as a Data Pipelines source.

2. Run the commands below to create a user named customerio.

    -- create a user named "customerio"
    CREATE USER customerio PASSWORD '<enter password here>';
   
    -- allows the "customerio" user to create new schemas on the specified database. 
    -- (This is the name you chose when provisioning your cluster)
    GRANT CREATE ON DATABASE "<enter database name here>" TO "customerio";
   

Add your Amazon Redshift source

As a part of this setup, you’ll provide Customer.io with user credentials that we’ll use to query your database. We recommend that you create a new user with Read Only access specifically for Customer.io, so you can manage Customer.io access to your database independent of any other Microsoft SQL users you have.

Your database must allow connections from 34.29.50.4 or 34.22.168.136 if your account is in our EU region. If our IP address is blocked, we won’t be able to connect to your database.

1. In the Data Pipelines tab, click Sources.

2. Click Add Source and select Amazon Redshift.

3. Provide your database information, including credentials to connect to your database, and click Save database.

   • The Name is a friendly name you’ll use to recognize your database whenever you reference it in Customer.io.
   • Enter Host address and the name of the database you want to connect to.
   • Enter a database user’s credentials and click Add database. We suggest that you create a service account and use service account credentials to set up your database.

     

     

4. Set up a Sync. A sync is the type of source data (identify, track, etc) you want to import from your database and click Next: Define Query. You can set up syncs for each type of data you want to import for your source.

   1. Provide a Name and Description for the sync. This helps you understand the sync at a glance when you look at your source Overview later.
   2. Select the type of data you want to import.
   3. Set the Sync Frequency, indicating how often you want to query your database for new data. You should set the frequency such that sync operations don’t overlap. Learn more about sync frequency.
   4. Select when you want to start the sync: whether you want to begin importing data immediately, or schedule the sync to start at a later date.

      

      

5. Enter the query that selects the data you want to import. See Queries below for more information about the information you’ll want to select for your sync. Click Run Query to preview results and make sure that your query selects the right information.

   

   

6. Click Enable to enable your sync.

Now you can set up additional syncs and connect your source to one or more destinations.

Adding syncs

After you set up your source, you can add additional syncs to import different types of data from your database. For example, you might want to import identify data for your users, and track data for their actions. Subsequent syncs can rely on your existing database, or you can add another database within your source.

1. In your source, go to the Syncs tab and click Add Sync.
2. Select your database or add a new one and click Next: Create Sync.
3. Set up a syncA sync is the type of source data (identify, track, etc) you want to import from your database. A sync is essentially the type of source call you want to make. and click Next: Define Query. You can set up syncs for each type of data you want to import.
   1. Provide a Name and Description for the sync. This helps you understand the sync at a glance when you look at your source Overview later.
   2. Select the type of data you want to import.
   3. Set the Sync Frequency, indicating how often you want to query your database for new data. You should set the frequency such that sync operations don’t overlap. Learn more about sync frequency.
   4. Select when you want to start the sync: whether you want to begin importing data immediately, or schedule the sync to start at a later date.

      

      
4. Enter the query that selects the data you want to import. See Queries below for more information about the information you’ll want to select for your sync. Click Run Query to preview results and make sure that your query selects the right information.

   

   
5. Click Enable to enable your sync.

Sync Frequency

You can sync data as often as every minute. However, we recommend that you set your sync frequency such that sync operations don’t overlap. If you schedule syncs such that a sync operation is scheduled to start while the previous operation is still we’ll skip the next sync operation.

Queries for each sync type

When you create a database sync, you provide a query selecting the people or objects you want to import, and respective properties. You’ll build your queries using the same principles from our Data Pipelines API.

For billing purposes, each row returned from your query represents an individual source operation/API call. Columns represent the traits or properties that you want to apply to the person, group, or event that your sync imports.

While we support queries that return millions of rows and hundreds of columns, syncing large amounts of data more then once a day can impact your account’s performance—including delaying campaigns or messages. When you set up your query, consider how much data you want to send and how often; and make sure you limit your results using the last_sync_time.

last_sync_time and limiting your results

Because we have to process each incoming row, and charge for each incoming row as an API call, it’s important to limit the amount of data you sync. You can do this by comparing timestamps against the last_sync_time value.

We expose last_time_sync as a Unix timestamp representing the date-time when the previous sync started. By comparing a timestamp against this value, you’ll only sync records that have changed since the last sync.

For your first sync, the last_sync_time is 0, so you’ll sync all records. After that, you’ll just get the changeset.

Identify

The identify method tells Data Pipelines who someone is and lets you assign unique traitsA key-value pair that you associate with a person or an object—like a person’s name, the date they were created in your workspace, or a company’s billing date etc. Use attributes to target people and personalize messages. Attributes are analogous to traits in Data Pipelines. to a person. Your query should compare a timestamp to the last_sync_time to ensure that you only import new data.

You can identify people by anonymousId and/or userId.

• anonymousId only: This assign traits to a person before you know who they are.
• userId only: Identifies a user and sets traits.
• both userId and anonymousId: Associates the data from the anonymousId with the person you identify by userId.

SELECT id AS userId, email_address AS email, fname, lname, msisdn AS phone
FROM users
WHERE last_updated >= {{last_sync_time}}

Track

The track method records things people do. Every track call represents an event.

You should track your audience’s activities with events both as performance indicators and so you can respond to your audience’s activities with campaignsA series of actions that you perform for each person who matches criteria. Campaigns typically send people a series of messages, but you can also use campaigns to send webhooks, update attributes, etc. in Journeys. For example, if your audience performs a Video Viewed or Item Purchased event, you might respond with other videos or products the person might enjoy.

Track calls require an event name describing what a person did. They must also include an anonymousId or a userId. Calls that you make with an anonymousId are associated with a userId when you identify someone by their userId.

Your query should compare a timestamp to the last_sync_time to ensure that you only import new data.

SELECT id AS userId, event_name AS event, products, total_price AS value
FROM events
WHERE timestamp > {{last_sync_time}}

Group

The Group method associates a person with a group—like a company, organization, project, online class or any other collective noun you come up with for the same concept. In Customer.io Journeys, we call groups objectsNot to be confused with a JSON object, an object in Customer.io is a non-person entity that you can associate with one or more people—like a company, account, or online course. You can use objects to message people based on changes to their company, account, or course itinerary..

Group calls require a groupId to represent the group. In almost every case, a group call should also include a userId to associate the person with the group. You can also include traits to provide additional information about the group (or the relationship between the person and the group). Find more details about the group method in our API specifications.

Your query should compare a timestamp to the last_sync_time to ensure that you only import new data.

Remember, group calls represent both an organization/group and relationships with users (by userId). Your query should include not only the groupId, but the userId so that you can capture relationships at your destinations.

SELECT companyId AS groupId, objectTypeId, companyname, employees, personId AS userId
FROM companies
WHERE last_updated >= {{last_sync_time}}

Page

The Page method records page views on your website, along with optional extra information about the page a person visited.

Your query should compare a timestamp to the last_sync_time to ensure that you only import new data.

SELECT id AS userId, metatitle as name, url, time_on_page
FROM pages
WHERE timestamp > {{last_sync_time}}

Screen

The Screen method sends screen view events for mobile devices. These help you understand the screens that people use in your app.

Your query should compare a timestamp to the last_sync_time to ensure that you only import new data.

SELECT id AS userId, screen_name as name, session_started
FROM screens
WHERE timestamp > {{last_sync_time}}

Alias

The Alias method combines two previously unassociated user identities. Some destinations automatically reconcile profiles with different identifiers based on whether you send anonymousId, userId, or another trait that the destination expects to be unique. But for destinations that don’t, you may need to send alias requests to do this.

In general, you won’t need to use the alias call; we try to handle user identification gracefully, so that you don’t need to merge profiles. But you may need to send alias calls to manage user identities in some destinations.

For example, in Mixpanel it’s used to associate an anonymous user with an identified user once they sign up.

SELECT id AS userId, old_id as previousId
FROM user_resolution
WHERE timestamp >= {{last_sync_time}}