Troubleshooting Tips for Data Pipelines
Pipelines are end-to-end integrations. When data doesn’t flow from your source to your destination, you may need to determine if you have a problem with your source, your destination actions, or your connection with your destination.
This page helps you check the various areas of your pipeline to find and solve problems.
General troubleshooting tips
- Check the Data Pipelines Connections page to determine if a pipeline has a problem. You can hover each source or destination to show pipelines the source or destination is a part of, including whether or not those pipelines have problems.
- Check your source’s Data In tab: are you sending the right calls to Customer.io?
- Check your destination’s Data Out tab: are you sending data to the destination?
Pinpoint the problem
Because Data Pipelines operates across your source and destination platforms, you may need to examine several places along your pipeline to find the issue.
- Is data reaching Customer.io? If not, the problem is in your source integration itself.
- Is data reaching your destination? If not, the problem may have to do with your destination’s credentials.
- If a source is connected to multiple destinations:
- Is the problem in a single destination? If yes, the problem is likely at the destination.
- Do all destinations have the same problem? If yes, your problem is likely in the source.
- If data reaches the destination, but doesn’t look quite right, then the problem is in your destination’s actionsThe source event and data that triggers an API call to your destination. For example, an incoming
identifyevent from your sources adds or updates a person in our Customer.io Journeys destination..
your pipeline?} a-....->|no|c(The problem is in
your source integration) a-->|yes|b{Does some data reach
the destination} e--->|yes|f(The problem is in
your source calls) b-...->|no|d(Check your destination's
credentials) b-->|yes|e{Does data fail at
multiple destinations?} e-.->|no|g{Is data missing entirely?} g-->|yes|h(The problem is a
destination action trigger) g-.->|no, data is partially
missing/wrong|i(The problem is a
destination action mapping)
Troubleshooting source problems
Go to your source page and check the Data In tab. You may want to perform some actions in your source to examine your incoming data:
- Do you see any data at all? If not, check your source credentials and make sure that you’ve added the correct library to your client, server, or app source.
- Do you see all the calls you expect to see? If not, you may need to make sure that your integration sends the correct methods—
identify,track, and so on. - Do your calls contain the correct data? If not, check that your source integration includes the correct information in the payload.
Data In tab) a-->b{Do you see any data?} b-...->|no|c(There might be an auth
problem in your source code) b-->|yes|d{"Is a call missing
(e.g. group, track)?"} d--->|yes|e(There's a call missing
in your source code) d-.->|no|f{Do calls contain
the right data?} f-.->|no|g(Update your calls to
send the right properties) f-->|yes|h{Are you sure you don't
have a destination problem?}
Troubleshooting destination problems
Go to your destination and check the Data Out tab and observe the data flowing to your destination:
- Do you see data at all? If not, make sure that your destination credentials are set up correctly.
- Do you not see data of a specific type? Check your destination actions to make sure that they’re triggered correctly.
- Do your actions not contain the correct data? You likely have an issue with the data mapped to one or more actions.
Data Out tab) a-->b{Do you see any data?} b-...->|no|c(Make sure your sources are
connected to the destination)-.->i(Check your
destination credentials) b-->|yes|d{Are you missing an action?} d---->|yes|e(Check that the action
trigger is correct) d-.->|no|f{Do calls contain
the right data?} f-..->|no|g(Make sure data is
mapped correctly) f-->|yes|h{Do you see failures
on the Overview tab?} h-->|yes|j(Your destination
service might be down) h-.->|no|k{Are you sure you don't
have a source problem?}
Use the Destination Tester to pinpoint issues
Your destination’s Tester tab can help you send test calls to find problems in your destination actions. When you go to your destination’s Tester tab, you’ll see a Test Action area. Here you can mimic source calls to test your destination actions.
For example, if you trigger an action when a Track Event Name is Clicked Button, you can set the Type to Track and change the event field to Clicked Button to test your action. If you see a response, your Trigger is set up correctly!
The Response area shows the call as sent to your destination. Here you can check that your action’s Mappings are set correctly. For example, if you map a property called firstName to fName, and fName is correctly set in your response, then your data is mapped correctly!
Troubleshooting destination actions
Destination actions tell us when to send data to your destination, and what data to send to your destination. That means that most problems with data reaching your destinations fall into two categories:
- The Trigger tells us when to send data to your destination
- The Mappings tells us what data to send to your destination
If you go to your destination’s Data Out tab, you can quickly see your problems. If you don’t see a particular action (and you know that the appropriate source data comes into your pipeline), there’s probably an issue with your trigger. If you don’t see specific data in an action, there’s an issue with your mapping.
Problems with triggers
A trigger determines when we send source data to your destination. If you’re using a default trigger, the trigger probably isn’t your problem!
But, if your action doesn’t trigger at all, or it does so inconsistently, then you likely have a trigger issue to troubleshoot.
But if you use a custom trigger, we occasionally see the following issues:
Some parameters aren’t compatible with each other. For example, if you set a trigger to fire when All conditions are true, and you set conditions where Type is Track and an Identify or Group Trait condition, your action will never fire. Track calls don’t contain identify/group traits, so your trigger conditions never occur together. This means that your action will never fire. When you add certain conditions, like Identify or Group Trait and Track Event Name, you’ll provide us the names/values of some of your source properties, and your spelling and capitalization matter! Make sure that you enter the names and values of trigger values exactly as they appear in your source calls. When you set up a trigger with multiple conditions, we default to All. This means that all your conditions have to be true to trigger the action. But if you want to perform the action when any of the conditions are true, you need to change the top-value under Trigger!Make sure your trigger conditions don’t conflict
Make sure you’ve spelled conditions correctly
Do you want to trigger when all conditions are true, or some conditions are true?
Problems with mappings
If data makes it to your destination, but it’s formatted incorrectly or missing specific properties, you probably have an issue with mappings!
When you map values to destinations, you often have to type out the names of properties. For each action, we look for properties exactly as you provide them. If the incoming property uses a different case or is spelled differently than your mapping, we’ll ignore it! If you map a value to your destination, but it isn’t always populated in your source call(s), it may be empty, null, or missing in your destination. You can set a fallback by using the Make sure you’ve spelled your property names and traits correctly
Set a fallback for empty values (using
coalesce)coalesce function in your mappings. If the value you want to map is empty or absent, we’ll use the coalesce fallback to make sure that the value is populated correctly.
What if I want to stop using Data Pipelines?
Data Pipelines is a series of sources that send data to destinations. To disable Data Pipelines, you’ll need to disable (or delete) all of your individual sources. This “turns off” Data Pipelines.
