This article demonstrates capturing real-time events from SmartPhones (iPhone, Android), injecting them in to InfoSphere Streams, and streaming the processed data out to web browsers where the data is rendered.

This demonstration can be extended to create applications that perform:

  • live polling
  • monitor your walking gait
  • monitor the acceleration of building’s elevators
  • verify¬† Galileo’s Tower of Pisa hypothesis – (voids SmartPhone’s warranty)

The effort involves locating,¬† configuring and accessing real-time data which are significant ‘barriers to demonstrate’.¬† The goal of this article is to remove the barriers by providing the components to stream live data.

Overview

The demonstration is depicted in the following diagram.


wsFlow

Diagram A.

The steps are :

  • SmartPhones’ accelerometers generates data.
  • HTML5 generates events in JavaScript.
  • Events transmitted, using JavaScript, via WebSocket protocol to InfoSphere Streams.
  • Events are received by¬† com.ibm.streamsx.inet.wserve on Streams toolkit.
  • The toolkit’s¬† WebSocketInject() operator injects data into stream.
  • Data is processed
  • The toolkit’s WebSocketSend(), operator sends the processed stream data via WebSocket protocol.
  • HTML5 receives events with the data from InfoSphere Streams.
  • The data is rendered by JavaScript utilizing¬†Smoothie Charts.

Resources

   Software Download links :

  Hardware : 

  • I’ve run this on VMWare hosted on PC (Workstation) as well as Mac (Fusion)
  • SmartPhone ¬†(sorry you’ve got to provide your own).¬† I’ve tried the application on the following hardware
    • IPhone4 IOS7 : Safari or Chrome for Mobile browsers
    • Nexus7/Android 4.4 ¬†: ¬†Chrome for Mobile

Components – provided

com.ibm.streamsx.inet.wsserve: Toolkit that implements WebSocket server protocol, within the streamsx.inet download.

com.ibm.streamsx.inet.rest : Toolkit that provides http access, located in the  streamsx.inet download.

basic.accelerometer : ReceiveToSend application provided in the streamsx.demo.accelerometer download.

ReceiveToSend.spl :  Streams application that utilizes com.ibm.streamsx.inet.wsserve toolkit. The application receives WebSocket messages in a XML format and transforms them into JSON messages as they are sent out.

index.html : Web page that acts as entry point to collect and display data. The web pages utilize the com.ibm.streamsx.inet.rest toolkit.

data/html/displayWebSocket.html :¬†This is accessed from index.html. It displays the current accelerometer and gyroscopic data. Selecting the ‘Blue’ button¬†will connect to server and start transmitting data (after putting up two alerts). It transmits XML messages to the ReceiveToSend.spl¬†via WebSocket protocol.

data/html/sinkWebSocket.html : This is accessed from the index.html page, receives WebSocket messages from the RecieveToSend.spl. The data is displayed in a table and scrolling graph.

Setup : Configuring VM’s network adapter

 

The Streams application gets data from a SmartPhone and displays a graphic on a web page. Both of these facilities need the IP address of the Streams instance running within the VM. Before you install the application you’ll¬† need to change the network configuration and get the IP address. If the network is not setup or the IP address is invalid, this demonstration will fail.

When using a VM for development by default you share the IP address with the host machine. Since the SmartPhone needs to communicate with the wsserve toolkit you’ll need to expose an IP address for the VM.

Configure the VM image so that the IP address is exposed on the network by setting the VM Network connection to ‘Bridged:..” Below is a screen capture of the settings VMWorkStation (win) and VMWare’s Fusion (mac) products.

Setup when hosted on Windows

Pasted_Image_3_3_14__10_18_AM

Setup when hosted on Mac

macConfig

NOTE : I have found that rebooting the host VM is necessary after making these changes. I never really ‘grok’ how the physical and virtual hardware interact.

NOTE:¬†This document assume that you are using Streams on the ‘vanilla’ QSE, if you are not or¬†encounter communications issues you will want to refer to ‘Appendix – Firewall configuration’.

Get the host’s IP address.

Get the IP address of where the Streams application is running. Use the ‘/sbin/ifconfig’ command to retrieve the IP address. Below is and example of running the command from the terminal.

ip

The IP address of the system that is hosting the application in this example is 9.72.139.152, we’ll use this value to connect from the SmartPhone and desktop browser to the Streams application.

NOTE: If you have a ‘ping’ application on your SmartPhone, you should attempt to ping the host IP address to verify connectivity.

NOTE : To quote one of the initial testers, “The biggest stumbling block is the network, WIFI, VM firewalls … many possible roadblocks since everyone’s environment is different”

Walk through

SmartPhones (iPhone,Android) have accelerometers that provide realtime positional data. Using  JavaScript and the wsserve toolkit you can stream data from these devices into Streams where it can be processed and displayed in realtime.

The schematic above (Diagram A) depicts the application, drilling into the application we see:

flow1

Diagram B.

Data from the SmartPhone(s)¬† flows into the left most operator WebSocketInject() and flows out the right most operator ‘ WebSocketSend()’.

This document walks through the configuring, installing and running of the ReceiveToSend application. The components of the application are:

  • HTML/JavaScript files that fetch the realtime data from the SmartPhone
  • Toolkits used to read/write web messages.
  • ReceiveToSend.spl that receives data consolidates/processes and sends it out.
  • HTML/JavaScript files that receives data via WebSockets and renders it as a graph.

Download and unpack

Download streamsx.inet application and streamsx.demo.accelerometer toolkit bundle using the links in the ‘Resources’ section above. After downloading, use the ‘Archive Manager’ to extract the files.

I downloaded and unarchived the files to the desktop.

Load the toolkits

Within the Streams explorer pane,¬† go to the ‘Toolkit Location’ element, right-click and select ‘Add Toolkit Location…’ set the directory to the location that the bundle was unpacked to. Here is a capture of my screen:

addToolkitLocation

Upon completion your ‘Toolkit Locations’ should include the imported toolkits as depicted below:

toollist

Import the Application

Import¬† project basic.accelerometer:¬† from the¬† ‘File’ menu ‘Import …’, browse to ‘Infosphere Streams Studio’ | ‘SPL Project’ where you will be prompted for the location of the project. Open the location where the application was unpacked, select the ‘Project Name’ of¬† ‘basic.accelerometer’, then select the ‘Finish’ button.

importApplication

Create the Application

To create (compile) the application in standalone mode: Expand ‘basic.accelerometer’ to ‘RecieveToSend’, right-click and select ‘New’ then¬† ‘Standalone Build’.

RH5V3_2_4

On the next page select ‘Apply’ and ‘Continue’ to create and build the application.

RH5V3_2

Run the Streams Application

After the ‘ReceiveToSend’ application builds, launch it:

RH5V3_2_1

We’re running the application standalone so that we can monitor the processing via the console. The running application ‘Console’ will look like:

RH5V3_2_5

The application is waiting for a web socket connection a well as  an HTTP connection, the HTTP communication; will occur on port 8080.

Connecting the SmartPhone(s) to Streams

We’re ready to connect to the Streams application with the SmartPhone. Using the IP address retrieved with the ‘/sbin/ifconfig’ command (above) connect to the application via the SmartPhone’s browser. In this example I’m connecting to host’s IP 9.72.139.152¬† port 8080 using a Nexus7.

Screenshot_2014-05-16-12-01-18

Select the blue ‘ Transmit Devices Accelerometer Data’ button to enter the data collector page, which looks like:

Screenshot_2014-05-16-12-01-41

On entry into the page the Alpha, Beta, Gamma, X, Y and Z values will be updating. If you set the device on flat surface the Beta will go to approximately zero.  Rotating it toward you, the Gamma will change.

Select the blue ‘Start Transmitting’ button to start sending data to the Streams application.

You’ll get two alerts as depicted in the next two screen shots.

Screenshot_2014-05-16-12-01-54

Select ‘OK’ to move on:

Screenshot_2014-05-16-12-02-04

Select ‘OK’ to move on.

The alerts are validating your hardware. Refer to the Appendix if you do not get the described alerts.

If you scroll down the page, you’ll find a ‘Communications’ section with a “Message;” indicating that the communications is “OPEN”.¬† The browser is now sending data to the Streams via WebSockets.

Screenshot_2014-05-16-12-19-29

Data flowing into Streams via WebSockets

On the ‘Console’ pane you’ll see data scrolling by. It will look something like:

runningConsole

Rotating the SmartPhone device, you should see the values in the windows change significantly.

Data flowing out of Streams via WebSockets

In order to see the data rendered using web sockets, you’ll need a bring up a current¬† version of Firefox or Chrome on a computer (laptop/desktop/SmartPhone). Use the same IP address/port you used to connect from the SmartPhone.

In this example I’m connecting to¬† host’s IP address (retrieved with the ‘/sbin/ifconfig’ command ) 9.72.139.152 port 8080 on a Mac using a Firefox browser, you should see something like:

macStart

Select the green ‘Display Streaming WS data’ button. You’ll be presented with the WebSocketServerSink messages:

display0

The three graphs will be scrolling with no data on them. This web page will connect to Streams using WebSockets and render the data streamed from Streams. The streamed data will be displayed on the three graphs and the table below the graph.

Select the green ‘Start Receiving’ button in order to setup the connection to Streams.

You’ll get two alerts as depicted in the next two screen shots.

display1

Select’OK’ to move on.

display2

Select ‘OK’ to move on.

The live graph at the bottom of the screen will display a line. That is the data from the device after it’s routed through Streams. If you rotate the SmartPhone the Beta value (second graph) will update in ‘near’ real time.

display3

The demonstration application will render data from up to nine SmartPhones. Every time a new connection is established between a SmartPhone and Streams a new line appears on the graph.¬† Connecting a second SmartPhone using the procedure described in section ‘Connecting the SmartPhone(s) to Streams’¬† will result in something like:

display4

Details

For details regarding the application and toolkits refer to the SPLDOCS accompanying the components.




 

Appendix – Issues

Error CDISP0127E ERROR : The following toolkit file is out of date …. :

This error may be encountered during the import or build process, this fix requires that rebuild the inet toolkit. The error will appear on the console, it will look like:

CDISP0127E ERROR: The following toolkit file is out of date: /home/streamsadmin/Desktop/com.ibm.streamsx.inet/toolkit.xml. This file is newer: /home/streamsadmin/Desktop/com.ibm.streamsx.inet/com.ibm.streamsx.inet.http/HTTPGetStream/HTTPGetStream.xml.

To fix:

  • Open an ‘xterm’¬†by double clicking on ”Terminal for streamsadmin’ .
  • Rebuild the toolkit by executing¬† the the following command in the xterm:
spl-make-toolkit -i <directory>

Where <directory> is the directory containing the toolkit.xml, as shown in the CDISP0127 error message. For example, in the error message above¬†the directory is ‘/home/streamsadmin/Desktop/com.ibm.streamsx.inet’, the command to issue is :

spl-make-toolkit -i /home/streamsadmin/Desktop/com.ibm.streamsx.inet

The following screen capture illustrates:

image7

After rebuilding the toolkit you should clean and rebuild the application, this is accomplished using the ‘clean..’ command within the ‘Project’ menu, it¬†looks like:

image8

Make sure that the ‘Clean all projects’ is selected and then hit ‘OK’.

After the application rebuilds you can proceed on to ‘Run the Streams Application’.


 

Hardware Validation Failure :

The code validates the hardware and software capabilities. If issues arise the following messages can appear.

issue1

This is indicating that your device does not have an accelerometer that is accessible from the browser. You may want to try another browser and/or upgrade to the latest version. I’ve used this software on the Chrome and Safari browsers and have reports that it works on Firefox as well.

issue2

This is indicating that the browser does not have WebSocket access.  You should upgrade to the latest version of the browser and/or try another browser.


 

java.net.BindException error:

If you inadvertently start a second application before the first stops you’ll get the following error in the console.

RH5V3_2

Only one application can allocate the socket at a time. To remedy the situation kill the application¬† by selection the ‘red box’ of the active application’s console.

RH5V3_2_1

 



 

Appendix – Firewall configuration

Contributed by Edward Pring (pring@us.ibm.com) – thanks

The Streams ReceiveToSend.spl application uses three network ports to communicate with the SmartPhone:

  • TCP port 8080 is used by the SmartPhone and browser to get HTML, CSS, and JavaScript files from the Streams application.
  • TCP port 8084 is used by the SmartPhone to send accelerometer data to the Streams application.
  • TCP port 8085 is used by the Streams application to send chart data to the browser

There may be one or more firewalls within the machine running Streams that could block any or all of these network ports. The instructions below can help you configure these firewalls to allow the SmartPhone and browser to communicate with Streams without compromising the security of the machine.

Linux ‘iptables’ firewall

The RHEL or CentOS operating system contains a firewall service called ‘iptables’. By default, it will block access to the TCP ports used by the Streams application. To allow the SmartPhone and browser to access them, do this:

  • Log into RHEL or CentOS as ‘root’.
  • Launch the Linux ‘Firewall Configuration’ application by selecting ‘System –> Administration –> Firewall’ from the Gnome menu, like this:

Image

 

  • Alternatively, Launch the Linux ‘Firewall Configuration’ application by typing ‘system-config-firewall’ at a Linux command prompt.
  • In the ‘Firewall Configuration’ window, select ‘Other Ports’ and then click ‘Add’.
  • In the ‘Port and Protocol’ window, select ‘User Defined’, enter ‘8080’ in the ‘Port / Port Range’ field, select ‘tcp’ in the ‘Protocol’ field, and click ‘OK”, like this:

Image

  • Repeat the previous step for ports 8084 and 8085.
  • Close the ‘Firewall Configuration’ application.
  • Confirm that ports 8080, 8084, and 8085 are enabled by typing ‘iptables -L -n’ at a Linux command prompt. Its output should include three lines like these:

ACCEPT tcp — 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8080
ACCEPT tcp — 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8084
ACCEPT tcp — 0.0.0.0/0 0.0.0.0/0 state NEW tcp dpt:8085

  • Alternatively, confirm that ports 8080, 8084, and 8085 are enabled by examining the file ‘/etc/sysconfig/iptables’. ¬†It should contain three lines like these:

-A INPUT -m state –state NEW -m tcp -p tcp –dport 8080 -j ACCEPT
-A INPUT -m state –state NEW -m tcp -p tcp –dport 8084 -j ACCEPT
-A INPUT -m state –state NEW -m tcp -p tcp –dport 8085 -j ACCEPT

VMware Fusion NAT firewall (for shared IP address)

If you are running RHEL or CentOS in a VMware Fusion virtual machine on Mac OS X, you may have configured its virtual ethernet adapter to ‘Share with my Mac’, like this:

Image2

In this configuration, the virtual ethernet adapter will use the IP address assigned to your real ethernet adapter, and VMware Fusion will place a NAT firewall between them, which will block access to the TCP ports used by the Streams application. To allow the SmartPhone and browser to access them, do this:

  • Edit the VMware Fusion NAT configuration file. This requires Mac OS X ‘Administrator’ privilege, which you can obtain temporarily by executing an editor with the ‘sudo’ prefix, like this:
sudo emacs /Library/Preferences/VMware\ Fusion/vmnet8/nat.conf
  • Enter your password for a Mac OS X password when prompted.
  • Near the bottom of the ‘nat.conf’ file, in the ‘[incomingtcp]’ section, insert one line for each TCP port used by the Streams application, following the pattern shown in the comments, like this:
[incomingtcp]
 # Use these with care - anyone can enter into your VM through these...
 # The format and example are as follows:
 #<external port number> = <VM's IP address>:<VM's port number>
 8080 = 192.168.199.127:8080
 8084 = 192.168.199.127:8084
 8085 = 192.168.199.127:8085
  • Save your changes to the ‘nat.conf’ file and exit from the editor.
  • Suspend your RHEL or CentOS virtual machine, close VMware Fusion.
  • Open VMware Fusion and resume your RHEL or CentOS virtual machine.

 VMware Fusion (for separate IP address)

If you are running RHEL or CentOS in a VMware Fusion virtual machine on Mac OS X, you may have configured its virtual ethernet adapter for ‘Bridged Networking’, like this:

Image3

In this configuration, your virtual ethernet adapter will be assigned a separate IP address than your real ethernet adapter, and VMware Fusion will not use a NAT firewall.

Mac OS X firewall

If you are running VMware Fusion in Mac OS X, then the Mac OS X firewall may block access to the ports used by the Streams application. To allow the SmartPhone and browser to access them, do this:

  • In the ‘System Preferences’ window, launch the Mac OS X ‘Security & Privacy’ application, like this:

Image4

  • In the ‘Security & Privacy’ window, select the ‘Firewall’ tab and unlock it by clicking the Lock icon in the lower left corner, like this:

Image5

  • Click “Firewall Options…”, then click “+”, add “VMware Fusion.app”, and select ‘Allow Incoming Connections’, like this:

Image6

  • Click ‘OK’ and close the ‘Security & Privacy’ application.

Linux ‘/etc/hosts’ file

Streams requires that the hostname and IP address of the RHEL or CentOS machine match. ¬†If these configuration changes to any of the firewalls results in a different IP address, you may have to change the ‘/etc/hosts’ file accordingly. ¬†To ensure that the hostname and IP address of the machine match, do this:

eth1      Link encap:Ethernet HWaddr 00:0C:29:D0:36:6D 
          inet addr:192.168.199.127 Bcast:192.168.199.255 Mask:255.255.255.0
          inet6 addr: fe80::20c:29ff:fed0:366d/64 Scope:Link
          UP BROADCAST RUNNING MULTICAST MTU:1500 Metric:1
          RX packets:2597 errors:0 dropped:0 overruns:0 frame:0
          TX packets:2554 errors:0 dropped:0 overruns:0 carrier:0
          collisions:0 txqueuelen:1000 
          RX bytes:507712 (495.8 KiB) TX bytes:519181 (507.0 KiB)
  • Edit the ‘/etc/hosts’ file. ¬†This requires Linux ‘root’ privilege, which you can obtain temporarily by executing an editor with the ‘sudo’ prefix, like this:
sudo emacs /etc/hosts
  • Enter the ‘root’ password when prompted.
  • Alter the ‘/etc/hosts’ file, if necessary, to map the IP address of the ethernet adapter to the hostname, like this:
127.0.0.1 localhost localhost.localdomain localhost4 localhost4.localdomain4
::1 localhost localhost.localdomain localhost6 localhost6.localdomain6
192.168.199.127 centos65.localdomain centos65.local centos65
  • Save your changes and close the editor.

 


 

Appendix – Extended Resources

The Web community has developed the technology that I’ve utilized to deliver realtime data.

  • HTML5 provides access to underlying hardware on your SmartPhone. Data includes: accelerometers, geolocation, ¬†camera, lightsensor, ¬†gyroscope, compass, battery status and¬†proximity events.
  • WebSockets is a light weight W3C specified protocol for low latency ¬†asynchronous communication between nodes. Sending data from¬†the SmartPhone hardware turns it into a participant in the ‘Internet¬†of Things’.

These resources are bundled in the SmartPhone’s browser(s). It is not necessary to install any software on the SmartPhone in order to run.

The following open source components have been utilized for this demonstration.

These components are bundled with the demo.

12 comments on"Streaming Realtime SmartPhone data with InfoSphere Streams"

  1. Kmell Van Hellsing August 17, 2014

    I can’t wait to do this lab. Thank you.

  2. Kmell Van Hellsing August 18, 2014

    Greetings.
    Here’s my feedback.
    After loading the Toolkit I couldn‚Äôt build the StandAlone Application. An error appeared telling that the ‚Äútoolkit.xml‚ÄĚ file (located in ‚Äúcom.ibm.streamsx.inet‚ÄĚ) was out of date. So I opened it (toolkit.xml) and saved it in order to change its modification date, and the problem was solved.

    I hope it could be useful for someone else. Thank you.

  3. Kmell Van Hellsing August 18, 2014

    Greetings. Here’s my feedback.

    When I tried to build the StandAlone Application I received this error:

    ‚ÄúLab/V121/streamx.inet.toolkits-1.2.1/com.ibm.streamsx.inet/impl/nl/include: No such file or directory‚ÄĚ

    I checked, and I found that the folder structure was correct. I think it was caused by the name of the folder I used to store the toolkit. I named it ‚ÄúSmartPhone Lab‚ÄĚ — with a space between the words — but in the description of the error the word ‚ÄúSpartPhone‚ÄĚ wasn‚Äôt shown.

    So, after erasing the space between the words in the name of the folder… the Standalone App was created. Yeah !

    I hope it could be useful for someone else.

    Thank you.

  4. Hi, I am getting the following error:

    CDISP0127E ERROR: The following toolkit file is out of date: /home/streamsadmin/Desktop/com.ibm.streamsx.inet/toolkit.xml. This file is newer: /home/streamsadmin/Desktop/com.ibm.streamsx.inet/com.ibm.streamsx.inet.http/HTTPPost/HTTPPost.xml.

    Anybody knows what is the problem?

  5. Kmell Van Hellsing August 18, 2014

    Thank you very much for sharing
    I completed the lab and tested it in a MOTOROLA RAZR MAXX with Android 4.1.2. using Chrome Explorer. Everything worked as described in the article YEAHHHHHHHH!

  6. Thanks Kmell. The error message that I get is in Streams. I would like to know how I can fix the CDISP0127E ERROR?

  7. There is a section in the article (above) which shows how to fix this problem (Section “Appendix ‚Äď Issues”). Thanks.

  8. Hi

    I am getting the following error.
    —- SPL Build for project basic.accelerometer started —- March 31, 2015 10:16:32 AM EDT
    Building toolkit index
    Building main composite: basic.accelerometer::ReceiveToSend using build configuration: Standalone
    /opt/ibm/InfoSphere_Streams/4.0.0.0/bin/sc -M basic.accelerometer::ReceiveToSend –output-directory=output/basic.accelerometer.ReceiveToSend/Standalone –data-directory=data -T -a -t /home/streamsadmin/Downloads/com.ibm.streamsx.inet: –no-toolkit-indexing –no-mixed-mode-preprocessing
    basic.accelerometer/ReceiveToSend.spl:2:5: CDISP0760W WARNING: The com.ibm.streamsx.inet.wsserver namespace is not defined in the use directive.
    basic.accelerometer/ReceiveToSend.spl:52:75: CDISP0053E ERROR: An unknown identifier was referenced in the SPL program: WebSocketInject.
    basic.accelerometer/ReceiveToSend.spl:73:17: CDISP0053E ERROR: An unknown identifier was referenced in the SPL program: WebSocketSend.

    CDISP0092E ERROR: Because of previous compilation errors, the compile process cannot continue.

    —- SPL Build for project basic.accelerometer completed in 2.799 seconds —-

    I initially had the problem with the error
    CDISP0127E ERROR: The following toolkit file is out of date: /home/streamsadmin/Desktop/com.ibm.streamsx.inet/toolkit.xml. This file is newer…

    And the steps above sorted it out. Anyone know what I should do now?

    Many thanks in advance

    MAL

  9. Hi! How can I send this data to an android application instead?

  10. Martin Siegenthaler April 09, 2018

    Hi:
    Developing this demo I used Chrome to send and display the data, I believe someone was able to run the demo connecting from an Android device using the Chrome browser. I’ve not developed an Android application (lack of hardware). This demo sends data to Streams.

    Sending data to Android would entail bringing up a web socket server on the Android. You’ll want to bring up webserver and a web socket server since you’ll need to download html page that connects to web socket. The html code provided in the demo maybe a good starting point.

    I would use the simulator to get the all the communication working before switching to phone data, it works on a desktop browser. Getting through the firewalls (every site has it’s idiosyncrasies) is a pain.

    Martin

    ps – You may want to verify that the browser does have access to the hardware data you want – due to ‘security’ issues some avenues to data have been removed.

Join The Discussion