Using and creating custom connectors

The Simple Data Pipe uses source-specific connectors to load data and store it as JSON in the Cloudant staging database. Connector data flow A connector accesses data through the source’s public APIs, authenticating with user-provided OAuth credentials. Connectors can optionally enrich the retrieved data, for example by adding tone analysis results or weather data.

Connector repositories on GitHub

The Simple Data Pipe application is installed with connectors for Salesforce and Stripe by default. You can manually deploy additional connectors for the following data sources by following the instructions in the next section:

Data Source Requires manual deployment? Data Enrichment Services
FlightstatsBETA Yes Weather Company Data
RedditBETA Yes Watson Tone Analyzer
RunkeeperBETA Yes No
SlackBETA Yes No
Sugar CRM Yes No
TrelloBETA Yes No

Deploy a connector

To add a custom connector in Simple Data Pipe

  1. Deploy the Simple Data Pipe application to Bluemix.
  2. If needed, create any custom service instances that are required by the connector.
  3. Add the desired connector repository as a Simple Data Pipe dependency.
  4. Restart the Simple Data Pipe application.
To learn more about the deployment process, refer to the detailed instructions in the connector’s README file (click a data source name, to open the connector’s readme).

Build a custom connector

If you want to load data from a cloud data source that doesn’t yet have a connector, you can build your own. We created template repositories on GitHub to help you hit the ground running. Choose one of the boilerplates and follow the referenced tutorial instructions:
Data Source Use Case OAuth support
Generic Be creative! Yes
RedditBETA Analyze Ask Me Anything comments for a specific topic Yes
StripeBETA Analyze your billing information Yes
YahooIn plan Fantasy sports analysis, anyone? Yes

Use cases

Related blog posts