In this post I will summarize the latest work being done in the github organization IBMStreams. As of today there are 16 toolkit repositories within the IBMStreams organization. There is a samples repository, 2 demo repositories and a utilities reposository. This has exceeded our initial expectations and continues to grow. Get involved and contribute, try out the toolkits or one of the samples, we need people to use these and provide feedback. If you find any problems open an issue. If you have ideas for samples, operators or new toolkits open an issue in the administration repository. If there are enhancements to an existing repository you want, open an issue against that repository.
The following are the latest repositories that provide new integrations for Streams:
- streamsx.mongoDB – Provides support for MongoDB.
- streamsx.thrift – Provides thrift server and client functionality
- streamsx.hbase – Read and write tuples to Apache HBase
- streamsx.document – Extract text and metadata from documents that are in a binary format.
These repositories were based on existing function in Streams and have been updated and extended.
The website for this toolkit is here. This contains the SPLDoc for the Internet toolkit. Additions to the toolkit include:
- com.ibm.streamsx.inet.http — Operators that interact with external HTTP servers.
- com.ibm.streamsx.inet.rest — Operators that embed an HTTP server to provide REST style API to streaming data
- com.ibm.streamsx.inet.wsserver — Operators that embed a WebSocket server to expose streaming data as WebSocket messages
This repository contains the operators that allow Streams to interact with an Hadoop distributed file system. These were migrated from the Streams BigData toolkit and updated. There are some samples included in the repository that demonstrate using the operators.
This has been updated to add support for Apache Kafka. Updates have been made to the MQTT, JMS and XMS operators. Looking for someone to add support for RabbitMQ — if you can help out go here. Do you need support for Kestrel? If yes, go here and indicated you need this support.
These provide new and improved capabilities. Feel free to open issues on these suggesting new functions or operators. We are always looking for contributions to add more capability to these toolkits.
Provides parsing and formatting for JSON data.
Provides operators and functions for parsing and formatting raw data.
Functions for getting the current time as well as conversion functions.
Contains various math based functions.
New toolkit repositories — looking for contributors
The following have been created and will have content coming soon. We need contributors to work on these toolkits.
Operators and functions to integrate Streams with social media sites.
OPerators and functions for analyzing log files.
Operators and functions for providing location based services.
Provides transportation based operators and functions.
Samples and Demos
There is a generic samples repository that currently contains 4 samples. This repository will be continually added to as new samples and examples are created. We are expecting to migrate and consolidate some existing samples into this repository over the summer months. The current samples are:
- streamsx.demo.accelerometer – A collection of demos streaming smartphones’ accelerometers / gyroscope data to InfoSphere Streams applications.
- streamsx.demo.logwatch – The purpose of the LogWatch family of applications is to teach concepts in SPL and Streams through a self-contained, small application with a well-defined problem statement.
The benchmark repository contains benchmarks used in the comparison of Streams and Storm.
The administration repository is used to manage IBMStreams on github. There is where new proposal are made, etc… If you want to contribute, have a proposal this repository is where to start.
In future posts we will provide details about the specific toolkits. In the meantime, please download the toolkits and try them out.