Using and creating custom connectorsThe Simple Data Pipe uses source-specific connectors to load data and store it as JSON in the Cloudant staging database. A connector accesses data through the source’s public APIs, authenticating with user-provided OAuth credentials. Connectors can optionally enrich the retrieved data, for example by adding tone analysis results or weather data.
Connector repositories on GitHubThe Simple Data Pipe application is installed with connectors for Salesforce and Stripe by default. You can manually deploy additional connectors for the following data sources by following the instructions in the next section:
|Data Source||Requires manual deployment?||Data Enrichment Services|
|FlightstatsBETA||Yes||Weather Company Data|
|RedditBETA||Yes||Watson Tone Analyzer|
Deploy a connector
To add a custom connector in Simple Data Pipe
- Deploy the Simple Data Pipe application to Bluemix.
- If needed, create any custom service instances that are required by the connector.
- Add the desired connector repository as a Simple Data Pipe dependency.
- Restart the Simple Data Pipe application.
Build a custom connectorIf you want to load data from a cloud data source that doesn’t yet have a connector, you can build your own. We created template repositories on GitHub to help you hit the ground running. Choose one of the boilerplates and follow the referenced tutorial instructions:
|Data Source||Use Case||OAuth support|
|RedditBETA||Analyze Ask Me Anything comments for a specific topic||Yes|
|StripeBETA||Analyze your billing information||Yes|
|YahooIn plan||Fantasy sports analysis, anyone?||Yes|
- Billing (Stripe.com)
- CRM (Salesforce.com)
- Sentiment Analysis (reddit data with Apache Spark and Watson)
- Flight stats analysis