IBM Streaming Analytics service – Improved Python Support

The combined release of the latest IBM Streaming Analytics service on IBM Bluemix and version 1.6 of the Python Application API introduces a suite of Python language features that greatly ease the development of streaming applications in the cloud. Applications developed with Python 3.5 can now be directly submitted from Python to a Streaming Analytics service – this can be achieved without the presence of a local IBM Streams install.

You can also create streaming applications in Python right from your browser with the IBM Data Science Experience (DSX)!  The DSX platform has been upgraded to integrate with the Streaming Analytics service, making it even easier to perform analysis on real time data.

Furthermore, Python developers can monitor submitted jobs by using the Python REST API. This is of particular interest to developers who seek to retrieve streaming data for purposes of visualization in Jupyter notebooks, debugging, or extra logging. A user can also invoke the REST API to cancel a previously submitted job; this provides a programmatic alternative to using the web console or the streamtool command.

 

Submitting an application to a Streaming Analytics Service

Running a Streams application in the cloud provides a number of conveniences compared to running locally; chiefly, that there is no need to maintain local hardware. A Streams application running in the cloud can more easily be scaled out in the event of high traffic, or throttled to be more cost efficient. Previous versions of the Python API have supported running locally through the STANDALONE and DISTRIBUTED execution contexts. In version 1.6, the list has been expanded to include the STREAMING_ANALYTICS_SERVICE  context, which executes an application within a Streaming Analytics service upon submission.

The following code illustrates how you can submit a Bluemix application:

from streamsx.topology.topology import Topology
from streamsx.topology.context import submit, ConfigParams, ContextTypes
top = Topology("Hello World Application", namespace="Jupyter")
hw_stream = top.source(["Hello", "world!"], name="Hello_World_Source")
hw_stream.print()
config = { ConfigParams.VCAP_SERVICES: "/home/streamsadmin/vcap.json",
   ConfigParams.SERVICE_NAME:"Streaming Analytics-be",
   ConfigParams.FORCE_REMOTE_BUILD : True}
r = submit(ContextTypes.STREAMING_ANALYTICS_SERVICE, top, config = config)

 

Output:

...INFO: Streaming Analytics service (Streaming Analytics-be): submit job response: {"artifact":"3165","jobId":"10","application":"Jupyter::HelloWorldApplication","name":"Jupyter::HelloWorldApplication_10","state":"STARTED","plan":"Standard","enabled":true,"status":"running"}

 

You’ll notice that the Bluemix connection information was saved in the /home/streamsadmin/vcap.json file – this is one of several ways in which credentials may be provided. For more information, see our application guide.

After the application has been compiled and submitted, its status and output are reflected in the Streams console, as shown in the following image of the web console:

The application, as displayed in the Streams console

Creating, building, and submitting a cloud application in five lines of Python code is a significant improvement in usability. Even more significant is the ability to do so without requiring a local Streams installation. This is handled automatically when using the STREAMING_ANALYTICS_SERVICE context if a Streams install is not present, or when the FORCE_REMOTE_BUILD option is specified.

As the application was built remotely, there is no local output regarding compilation. In the event that an error occurs during build, the output will be retrieved and displayed.

The Python REST API

Streaming applications are unusual in that the amount of data they process is unbounded, and as such the application is treated as if it never terminates. Therefore, it becomes necessary to monitor a job while it is running, and to observe metrics such as congestion, flow rate, or even the individual tuples present on a stream. To make such monitoring easy to Python developers, V1.6 of the Python API includes the Python REST API which allows an authenticated user to observe Streams resources at runtime, and to cancel jobs when needed.

For example, a user might write a script that checks the status of a service instance before submitting an application:

from streamsx import rest

sc = sc=rest.StreamingAnalyticsConnection(vcap_services="/home/streamsadmin/vcap.json", service_name="Streaming Analytics-be")
instance = sc.get_instances()[0]
print(instance.status)

Output:

running

The graphical representation of data is central to modern data science and engineering; therefore, the REST API also supports obtaining individual tuples from a view to visualize a stream’s contents. For example, a user can compare an engine’s temperature against its likelihood of failure in real time in a Jupyter notebook:

from streamsx import rest

sc = rest.StreamingAnalyticsConnection(vcap_services="/home/streamsadmin/vcap.json", service_name="Streaming Analytics-be")

instance = sc.get_instances()[0]
view = instance.get_views()[0]

fig, ax = create_plot([], [], title="Engine Temp Vs. Probability of Failure", xlabel = "Probability of Failure",
ylabel = "Engine Temp in Degrees Celsius")
xdata = []
ydata = []

try:
  queue = view.start_data_fetch()

  for line in iter(queue.get, None):
   ydata.append(float(line))
   xdata.append(len(ydata))
   ax.lines[0].set_xdata(xdata)
   ax.lines[0].set_ydata(ydata)
  fig.canvas.draw()
except:
   raise
finally:
   view.stop_data_fetch()

An image of real time data being visualized in a Jupyter notebook

Lastly, more than simply monitoring applications, the Python REST API provides a mechanism for canceling remote jobs.

from streamsx import rest

sc = rest.StreamingAnalyticsConnection(vcap_services="/home/streamsadmin/vcap.json", service_name="Streaming Analytics-be")
inst = sc.get_instances()[0]
job = inst.get_jobs()[0]
canceled = job.cancel()

if canceled:
  print(“The job has been canceled”)

Output:

 

The job has been canceled

 

Get Started

Ready to try out the new features? You can:

Additional Information

1 comment on"What’s new in the Streaming Analytics service on Bluemix"

  1. Great article Will, thanks for the explanations of how to write streaming apps using pylhon!

Join The Discussion