Written by Samantha Chan.
When developing an application that handles geospatial data, it is imperative that we can actually view the data on a map, and see how it is being manipulated by the application.
In this article, I am going to demonstrate how you can easily write a Streams application to display your application Geospatial data onto a map in a web browser.
MapViewer Sample
The MapViewerSample from the Github Samples repository is developed to show how this can be done. The sample application randomly generates location data for two types of entities around Hong Kong Island. Taxi data will be displayed with a red marker on a map. Public Minibus data will be displayed with a green marker. Here’s a screen capture that shows the end result of our sample application:
The code for this sample application can be found here: https://github.com/IBMStreams/samples/tree/main/Geospatial/MapViewerSample
Building Blocks
The following diagram shows the major building blocks of the the MapViewerSample application:
The application contains the following components:
- A data source that produces geospatial data to be displayed on a map. In the case of the MapViewer sample, we are simply generating random data points around Hong Kong. In a real application, this can come from GPS data from mobile phones, smart cars, etc. The data source, at a minimum, needs to provide the following information:
- Entity ID – ID of the entity to be displayed on the map
- Location data – in the form of Well-Known Text
- HTTPTupleView operator – The HTTPTupleView from the com.ibm.streamsx.inet toolkit provides the following capabilities in the application:
- Ability for us to query for tuple data feeding into the HTTPTupleView operator via Http REST API.
- Embedded Jetty server – When the operator is started, a Jetty server is started that allows us to query for tuple data. In addition, the Jetty server can also serve static HTML files.
- map.html – This is the static HTML file to be served by the embedded Jetty server. map.html contains Javascript that periodically queries for latest tuple data from HttpTupleView using the Http REST API. Then it uses OpenLayer APIs to parse the tuple data, and display the location data on a map.
Data Source and the Main application
This is the SPL code from the Main composite of the sample application. This application uses a Custom operator to randomly generate location data. The custom operator generates the following information:
- Id of the entity – randomly generated based on number of entities required.
- Latitude – randomly generated based on a set min and max values
- Longitude – randomly generated based on a set of min and max values
With the latitude and longitude information, the application calls the com.ibm.streams.geospatial.ext::point(…) function to convert the coordinates to a WKT string. The tuples are submitted to a composite operator called MapViewer.
/** Main composite application to generate location data and submit to MapViewer.
* Each tuple has the following attributes:
* * id - id of the entity to be displayed
* * wkt - geometry of the entity specified as WKT string
* * updateAction - tells the Javascript how to update the entity. 1 to add or update, 0 to remove.
* * note - additional information tagged with the entity. Note is displayed as a popup on the map.
* * markerType - allows us to control the color of the entity markers on the map.
*/
composite Main
{
graph
(stream<rstring id, rstring wkt, uint32 updateAction, rstring note, MARKER_TYPE markerType> GreenEntities_out0) as GreenEntities = Custom()
{
logic
onProcess :
{
while(! isShutdown())
{
// Randomly generate entities in Hong Kong
float64 latitude = randomLatitude(22.248429, 22.282425) ;
float64 longitude = randomLongitude(114.137821, 114.236355) ;
rstring id = randomId(10, 10) ;
// convert coordinates to wkt string
rstring wktGeometry =point(longitude, latitude) ;
// submit to MapViewr, set upactionAction to 1 to add point to the map
submit({ id = id, wkt = wktGeometry, updateAction = 1u, note =
"Public Mini Van: " + wktGeometry, markerType = GREEN },
GreenEntities_out0) ;
block(0.01) ;
}
}
}
() as MapViewer_2 = MapViewer(RedEntities_out0, GreenEntities_out0)
{
}
}
MapViewer Composite
The most important part of this application is the MapViewer composite. This composite operator wrappers a HTTPTupleView operator and sets up the resource and context paths for the Jetty server.
public composite MapViewer(input stream<MapViewerT> In0 )
{
param
expression $windowSize : 1 ;
graph
() as data = HTTPTupleView(In0 as inPort0Alias)
{
window
inPort0Alias : sliding, count($windowSize), count(1), partitioned ;
param
contextResourceBase : getThisToolkitDir() + "/etc" ;
context : "map" ;
partitionKey : "id" ;
config
placement : partitionColocation("mapviewer") ;
}
}
The MapViewer composite expects schema of the input stream to be of type MapViewerT. MapViewerT is defined as follows:
/** The expected schema for the MapViewer composite
* id - the id of the entity
* wkt - geometry of the entity specified in WKT format
* updateAction - 1 to add / update the entity in the map, 0 to remove the entity from the map.
* note - information to be displayed in the popup
* markerType - if the entity is a point, the color of the marker to use. MARKER_TYPE can be RED, GREEN OR YELLOW.
*/
type MapViewerT = rstring id, rstring wkt, uint32 updateAction, rstring note, MARKER_TYPE markerType ;
In the composite, the HTTPTupleView operator is set up with a sliding window, partitioned by the entity ID. This allows us to keep history for each of the entities in case we want to show where the entities have traveled over a period of time. The window size is configurable via the $windowSize parameter.
The HTTPTupleView operator is also set up with contextResourceBase and context parameter. The contextResourceBase parameter defines the directory location of resources that will be available through the URL context defined by the context parameter. In this case, it is expected that the static html resources are located in <Applicaton Directory>/etc directory.
The context parameter defines a URL context path that maps the resources defined by the contextResourceBase.
With this configuration, you can browse for static html files from the Streams application using the following URL: http://<PE host>:8080/map
map.html
Next, we need an HTML page for displaying the data. The map.html file is located in the MapViewerSample/etc/ directory. It contains JavaScript that displays a map, queries for data from the Streams application and displays the data as markers on the map.
To do this, here are some of the important parts of this html file:
- Import OpenLayer.js from www.openlayers.org – To have access to the OpenLayers Javascript APIs, import the following script:
<body>
<div id="test"></div>
<div id="map-canvas"></div>
><script src="http://www.openlayers.org/api/OpenLayers.js"></script>
<script>
:
:
- Set up OpenLayer map and layers in the initialize function. In this function, we set up three layers. At the base layer is a map canvas. The markers layers is for displaying any POINT data on the map. The polygon layer is for displaying any POLYGON data on the map. We then call “loadData” to get the data from the Streams application.
function initialize() {
var map_options = {
div : this.mapDiv,
allOverlays : false,
maxExtent : this.mapExtent,
controls : [ new OpenLayers.Control.DragPan(),
new OpenLayers.Control.Navigation(),
new OpenLayers.Control.PanZoomBar(),
new OpenLayers.Control.ScaleLine(),
new OpenLayers.Control.MousePosition(),
new OpenLayers.Control.LayerSwitcher() ]
};
map = new OpenLayers.Map('map-canvas', map_options);
map.addLayer(new OpenLayers.Layer.OSM());
// create marker layer
markers = new OpenLayers.Layer.Markers("Markers");
map.addLayer(markers);
// create polygon layer
vectors = new OpenLayers.Layer.Vector("Polygon", {
styleMap: new OpenLayers.StyleMap({
"default": new OpenLayers.Style({
fillColor: "#33CC00",
strokeColor: "#000000",
strokeWidth: 1
})
})
});
map.addLayer(vectors);
// retrieve data from HTTPTupleView
loadData();
}
- In the loadData method, the method constructs the URL for querying the latest data from input port 0 of the HTTPTupleView operator. The URL to use for fetching data is: http://[PE Host]:8080/map/data/ports/0/tuples
- Next, we construct an HTTP request with the constructed URL. When a response is received, we call the updateMap method with the response text from the request.
- At this time, we also schedule a timeout for the page. loadData will be called again after the timeout period to refresh the markers on the map.
function loadData () {
// retreive data from input port of HTTPTupleView
var url = window.location.protocol + "//" + window.location.host + "/map/data/ports/input/0/tuples";
// construct HTTP request and send
var markerReq = new XMLHttpRequest();
markerReq.open("GET", url, true);
markerReq.onreadystatechange = function() {
// when we get the response back, update marker and polygon locations
if (markerReq.readyState == 4 && markerReq.status == 200) {
updateMap(markerReq.responseText);
}
}
markerReq.onLoad = markerReq.send(null);
// refresh data every x second based on period parameter
timeoutID = setTimeout('loadData()', getPeriod());
}
- In the updateMap(response) method, the response text will be encoded as JSON. We will first parse the response and construct objects from the JSON text.
- Location data of the entity will be encoded as WKT. We will use the OpenLayers WKT formatter to parse the WKT string from tuple data. The formatter returns a Feature. We need to tranform the feature coordinate from WGS 1984 to Sperical Mercator Projection before it can be displayed on the map.
function updateMap(response) {
try {
// HTTPTupleView retuns tuples information as JSON, parse the JSON into a list of objects to process
var allObjects = JSON.parse(response);
// for each object, update marker or poly accordingly.
for (var i=0; i<allObjects.length; i++) {
var markerID = allObjects[i].id;
var wkt = allObjects[i].wkt;
var markerType = allObjects[i].markerType;
var updateAction = allObjects[i].updateAction;
// construct WKT formatter to parse out geometry from tuples
var formatter = new OpenLayers.Format.WKT();
// parse wkt string from tuples
var feature = formatter.read(wkt);
// transform from WGS 1984 to Spherical Mercator Projection
var transformedFeature = feature.geometry.transform(new OpenLayers.Projection("EPSG:4326"), map.getProjectionObject());
var vertices = transformedFeature.getVertices();
// this is a point, number of vertices must be 1
if (vertices.length == 1) {
:
:
:
- Next we will use the transformed coordinates and create a marker, and add the marker to the Marker Layer of the map.
// create longlat object for marker
var myLongLat = new OpenLayers.LonLat(transformedFeature.x, transformedFeature.y);
:
:
// if updateAction > 1, add marker to map
if (updateAction > 0)
{
var icon = new OpenLayers.Icon(getIcon(markerType));
var marker = new OpenLayers.Marker(myLongLat, icon.clone());
markers.addMarker(marker);
:
:
Putting This All Together
To view the location data on a map in a browser, you need to first launch your application to a Streams instance. Once the application is successfully started, start a web browser and enter the following URL:
http://[PE Host]:8080/map/map.html
What’s happening under the covers:
- When the Main SPL application is started, the HTTPTupleView operator starts an embedded Jetty server, waiting for HTTP requests.
- The Custom operator from the Main composite generates random location data and submits it to HTTPTupleView operator.
- The HTTPTupleView operator put the tuples into a sliding window. The window is partitioned by entity ID.
- When map.html is requested from a web browser, the Jetty server serves the static map.html page from the Streams application.
- When the web page is initialized, the java script loads the map onto the browser. Next it constructs an HTTP request with the following URL to get the latest data from the HTTPTupleView operator: http://[PE Host]:8080/map/data/ports/0/tuples.
- When the Jetty server receives the HTTP request, it processes the request, fetches the correct data and returns the latest data in the form of JSON.
- When map.html receives a response back, it decodes the JSON response. From the response, fetches and parses the WKT string into the correct coordinate system.
- For each tuple / entity received from the response, we add a marker to the Marker Layer on the map.
What about popup?
In the previous section, we mentioned that each of the entities can be tagged with a note. The note information can be displayed in popup on the map. To see this information, use the following URL:
http://[PE Host]:8080/map/map.html?popup=true
You should see the following in the web browser with the popup parameter enabled:
Running in Cloud Pak for Data
If you run this application in Streams for Cloud Pak for Data, you will need to make some changes because the HTTP endpoint used by the map port is not visible outside the Streams application pod.
So you need to set up the Streams endpoint monitory. This is a reverse proxy server that provides access to the HTTP endpoint that the Map viewer uses.
See the readme for details and setup instructions.
After you set up the reverse proxy, you need to update the map.html file to use the URL of the proxy server instead of directly connecting to the operator.
The HTTP endpoint has this prefix added by the proxy:
/streams/jobs/job-id/.Therefore, you will also need to update the URL used by the web page to add the prefix used by the proxy,
here.
var jobId = getJobId(window.location.pathname);
var url = "/streams/jobs/" + jobId + "data/ports/input/0/tuples";
Links
The code for this sample application can be found here: https://github.com/IBMStreams/samples/tree/main/Geospatial/MapViewerSample
Streams endpoint monitor
#CloudPakforDataGroup