Overview

Skill Level: Beginner

This recipe - co-authored with Julia Wiegel and Rene Meyer - shows how to ingest and process weather data using the Weather Company Data service (API), IBM Cloud Functions (based on Apache OpenWhisk) and IBM Event Streams (based on Apache Kafka).

Ingredients

Software Requirements

To obtain an IBM Cloud Account and get access to the IBM Cloud and to IBM Watson Studio, please follow the instructions outlined here:

Step-by-step

  1. Introduction

    This recipe shows how to ingest and process weather data coming from the Weather Company using IBM Cloud services. It covers 2 key steps: ingestion of weather data into the IBM Cloud using the Weather Company API and IBM Cloud Functions, and delegation of the captured weather data for further processing to IBM Event Streams in which IBM Event Streams takes the role as a generic message broker. Further processing and analysis of the weather data – either as data at rest or as data in motion (real time) – will be covered by other recipes and take IBM Event Streams as a base vehicle for reading events.

    The recipe will cover the following steps:

    • Introduction to the architecture
    • Getting started by instantiating the required services and installing the needed command line tools.
    • Overview of the Weather Data API.
    • Setting up IBM Function packages for the Weather Company Data and IBM Event Streams.
    • Ingestion of weather data using IBM Cloud Functions and the Weather Company Data service.
    • Sending of weather data to IBM Event Streams using IBM Functions.
    • Scheduling of IBM Functions using time based triggers.
    • Monitoring of IBM Functions.
    • Receiving weather data from IBM Event Streams using IBM Functions.
    • On-demand ingestion of weather data using the Weather Company API and IBM Functions.

     

    In this recipe we shall use IBM (public) Cloud for the exercise. However, insofar that IBM Functions is based on Apache OpenWhisk and IBM Event Streams on Apache Kafka, nothing speaks against deploying the same configuration either on premise or in a private Cloud such as IBM Cloud Private.

  2. Architecture Overview

    Before you start it is important to provide you with some details on the architecture of the solution. The architecture diagram is shown below and is a typical IoT architecture with services for event data ingestion, event handling (message bus), landing zone (object storage) and event data processing to transform, aggregate, load and visualize event data in a form so that it becomes business relevant. In other words, it is a data refine pipeline turning “raw oil” into “business relevant information‚ÄĚ covering data at rest.

    In this recipe we will focus on the first 3 services whereas the other services (rendered slightly transparent) will be covered by upcoming recipes:

    01.1-Architecture

    Weather data can be retrieved using either the Weather Company Data service on IBM Cloud or directly from the Weather Company using the API. In both cases the basis for retrieving the various kind of weather data is REST calls, regardless of wether the data to be retrieved is historic weather data, current weather data, weather prognosis data or special kind of data such as e.g. lightning data  Рjust to mention a few of the possibilities. The Weather Company Data service on IBM Cloud offers a subset of the data services available from the Weather Company but has the advantage that it can be used directly by instantiating the service in IBM Cloud, whereas the Weather Company API requires a specific API Key that need to be obtained from the Weather Company. We shall therefore start with the Weather Data service in this recipe.

    IBM Cloud Functions (based on Apache OpenWhisk) is a Function-as-a-Service (FaaS) platform which executes functions in response to incoming event. The actions for reading weather data events from the Weather Company Data service will be triggered by timers. The raw weather data will then be ingested, slightly processed upon which it will be send further onwards to IBM Event Streams. This means that the functions take the role as producers for event data seen from the point of view of the Event Stream component. Various kind of consumers can then read the weather data and process it further on either as data in motion in real time e.g. using Apache Spark Streaming, or as data at rest using e.g. the IBM Cloud Object Storage Bridge for storing the data in IBM Cloud Object Storage. From there Apache Spark or IBM SQL Query can then be used for Extracting, Transforming and Loading (ETL) the data e.g. into a file or a Data Warehouse. These aspect of processing and analyzing weather data will be covered by other recipes which will focus on services of the IBM Cloud data platform and IBM Watson Studio.

    There are several alternatives to the current architecture. One alternative would be to use Node.js applications (or applications defined in any other suitable programming language) to define the ingestion and delegation logic. The advantage of using Functions is 3 fold. First of all functions are minimal and allow the developer to focus on the logic solely since there is no overhead for setting up the application and compiling and deploying it. Secondly, functions are more cost effective since costs will only incur when the function is executing and will be determined on basis of the RAM and time used whereas applications incur costs even if they are idle waiting for events to occur. Third and last, IBM Cloud Functions are distributed and highly available Out-Of-The-Box whereas this would need to be achieved using applications e.g. by deploying the application in a Kubernetes cluster spanning several data centers (availability zone) within a region.

     

    Weather Company Data and API

    The Weather Company offers accurate, personalized and actionable weather data and insights via the Weather Company API’s, which provides information covering historical, current and forecast weather conditions. This information may in turn help companies optimize operations, reduce cost, improve safety, and uncover new revenue opportunities.

    The Weather Company retrieves its data from  a variety of resources including 250.000 personal weather stations (PWD) as well as e.g. 20 million smartphones (pressure data) and 50,000 flights per day (atmospheric data). A comprehensive set of APIs are available covering current weather conditions, weather forecast, severe weather conditions, ligthning detection, historical weather data as well as APIs for e.g. aggriculture, traffic and lifestyle. Beyond various forms of historic, current and forecasting data the API also provides current condition and forecasting images that can be used to visualize weather conditions:

    01.5-Weather-Data-API

    The Weather Company API is higly available and globally available with hight resolution. Current on Demand weather information (Enhanced Current Conditions) is for example available down to a resolution of 500 meters. A subset of the capabilities are available directly on IBM Cloud through the Weather Company Data service. This service lets you integrate weather data from the Weather Company into an IBM Cloud application. You can retrieve weather data for an area specified by a geolocation. The following set of information is provided by that service:

    • Hourly forecast: An hourly weather forecast for the next 48 hours starting from the current time, for a specified geolocation.
    • Daily forecast: A daily forecast for each of the next 3, 5, 7, or 10 days starting from the current day, including forecasts for the daytime and nighttime segments.
    • Intraday forecast: A daily forecast for each of the next 3, 5, 7, or 10 days starting from the current day, which breaks each daily forecast into morning, afternoon, evening, and overnight segments.
    • Current conditions: Observed weather data (temperature, wind direction and speed, humidity, pressure, dew point, visibility, and UV Index) plus a weather phrase and a matching weather icon.
    • Historical data: Observed weather data from site-based observation stations for a specified geolocation that includes current observations and up to 24 hours of past observations.
    • Weather alerts: Government-issued weather alerts, including weather watches, warnings, statements, and advisories issued by the National Weather Service (US), Environment Canada, and MeteoAlarm (Europe).
    • Location services: The ability to look up a location name or geocode (latitude and longitude) to retrieve a set of locations that match the request.
    • Almanac services: Historical daily or monthly weather data sourced from National Weather Service observations stations from a time period spanning 10 to 30 years or more.

     

    In this recipe we shall mainly use the Weather Company Data service available on IBM Cloud for two reasons: 1) it can be directly provisioned from the IBM Cloud catalog without any need to obtain an API Key from the Weather Company and 2) the service offers a free plan which comes in handy for experimental purposes. In this context we shall focus on current weather conditions for a geo location of your choice.

    In extend we shall also look into how to retrieve information using the Weather Company API. However, to avoid the need to request an API key, we will just introduce the API and its URLs in section 4 and then showpresent a generic solution in section 11 showing how IBM Functions can be used to ingest data from any kind of URL – be it an URL from the Weather Company Data service on IBM Cloud or an URL of the Weather Company API.

     

    IBM Cloud Functions

    IBM Cloud Functions executes functions in response to incoming events and costs nothing when not in use. Functions provide a platform allowing users to develop, run, and manage application functionalities without the complexity of building and maintaining an infrastructure typically associated with developing an application. Using function makes it possible to execute logic in response to database change, perform analytics on sensor input messages, provide cognitive computing via chat bots, schedule tasks performed for a short time and invoke auto scaled APIs and mobile back ends. The current architecture for processing weather will execute functions periodically by defining schedules following a cron-like syntax to specify when actions are supposed to be executed.

    01.2-Functions

    The Apache OpenWhisk platform supports a programming model in which developers write functional logic (called actions), in any supported programming language, that can be dynamically scheduled and run in response to associated events (via triggers) from external sources ( feeds) or from HTTP requests. The project includes a REST API-based Command Line Interface (CLI) along with other tooling to support packaging, catalog services and many popular container deployment options.

    Triggers are classes of events that may occur and can come from a variety of sources. Invocations of a trigger can be blocking, non-blocking, or periodic. Actions are code that runs in response to an event (i.e. they are event handlers), and can be written in a variety of languages, such as JavaScript, Python, Java, PHP, and Swift or any other language by packaging the code with Docker. Actions can be composed to create sequences that increase flexibility and foster reuse. By composing actions it is also possible to define conditionals, loops and error handling. Rules in turn provides an association of a trigger to an action in a many to many mapping. The concept of rules fosters reuse and modularity by allowing a single action to be associated with several triggers.

    Apache OpenWhisk makes it simple for developers to integrate their actions with many popular services using packages that are provided either as independently developed projects under the Apache OpenWhisk family or as part of the IBM default Catalog. Packages offer integrations with general services such as Kafka message queues, databases including Cloudant, Push Notifications from mobile applications, Slack messaging, and RSS feeds. Development pipelines can take advantage of integrations with GitHub, JIRA, or easily connect with custom data services from IBM Watson for Translation or Speech-to-Text, as well as the Weather Company. In this recipe we shall use the Alarms package to schedule times or recurring intervals to run a sequence of actions for reading weather data and delegate the data to Event Streams.

    The advantage of using IBM Cloud Functions are numerous:

    • Cost Effective: cost is only incurred during the short time that the function runs and is dependent on the RAM used.
    • Automatically scalable: action instances scale to meet demand exactly, then disappear.
    • Ease of integration: actions can be triggered from events in other services, or directly via REST API.
    • Polyglot: functions can be developed in one of the natively supported languages, or alternatively in any other language that can be deployed in a Docker container.
    • Highly Available: the current implementation of IBM Functions is transparently distributed over several Availability Zones in a single region which significantly increases the availability of the service, and thus also the availability of applications that rely on the use of this service.
    • Open Source: based on Apache OpenWhisk which avoids vendor lock in.
    • Flexible: Apache OpenWhisk can be deployed on premise, in a private cloud or in a public cloud.

     

    IBM Event Streams

    The IBM Event Streams (previously also named IBM Message Hub) service is based on Apache Kafka. Using Kafka we can achieve a decoupling of message producers and message consumers which results in larger flexibility compared to a direct one-to-one communication scheme. At the same time Kafka offers essential features such as

    1. Persistence of messages to avoid that messages are lost,
    2. Distributed and horizontal scaling allowing for multiple servers and consumers and last but not least
    3. Automatic recovery from broker failure.

     

    Kafka runs as a cluster of one or more (distributed) servers called brokers that store streams of records in categories called topics, where a topic is a category or feed name to which records are published by producers and read by consumers. The Kafka cluster is a collection of brokers distributed on one or more machines so that the load can be processed simultaneously. Zookeeper monitors the health and maintains the metadata information about the brokers. It is the central service for maintaining configuration information, monitoring overall health of the cluster and balancing broker assignments and re-assignments. For example, should a broker go down, Zookeeper will need to distribute its load to the remaining brokers.

    01.3-Event-Streams-Architecture

    Each record consists of a key, a value, and a timestamp. Topics in Kafka are always multi-subscriber; that is, a topic can have zero, one, or many consumers that subscribe to the data written to it. Topics are broken up into ordered commit logs called partitions. Each partition is an ordered, immutable sequence of records that is continually appended to. The records in the partitions are each assigned a sequential id number called the offset that uniquely identifies each record within the partition.

    01.4-Event-Stream-Topic

    The producers are in principle responsible for choosing the partition within the topic when sending a message.¬† Messages sent by a producer to a particular topic partition will be appended in the order they are sent, i.e. messages from the producers are appended to the end (right) of the partition. Consumers can decide whether they want to read from the beginning of the topic, or whether they want to read from the next record. Consumers can furthermore be grouped into consumer groups that will read all read messages from one and the same topic to achieve load balancing and higher throughput. Each record published to a topic is delivered to one consumer instance within each subscribing consumer group. Basically this is achieved by letting all consumers in a group share the notion of “next record to read” for each partition. The notion of topics and consumer groups allow Kafka applications to adopt a queue as well as a broadcasting protocol for communicating messages between producers and consumers. In the current architecture it gives us the flexibility to add consumers such as e.g. the IBM Cloud Object Storage Bridge for storing the weather data to IBM Cloud Object Storage as well as consumers for processing the data in real time using e.g. Apache Spark Streaming.

    Kafka provides fault tolerance and high throughput by distributing the partitions over the servers in the Kafka cluster and by replicating the partitions across a number of servers. Each partition has one server which acts as the “leader” and zero or more servers which act as “followers”. The leader handles all read and write requests for the partition while the followers passively replicate the leader. If the leader fails, one of the followers will automatically become the new leader. Each server acts as a leader for some of its partitions and a follower for others to balance load across the cluster. Moreover, Kafka messages are kept in persistent store for a period of time. The Kafka cluster retains all published records‚ÄĒwhether or not they have been consumed‚ÄĒusing a configurable retention period. For example, if the retention policy is set to two days, then for the two days after a record is published, it is available for consumption, after which it will be discarded to free up space. Kafka’s performance is effectively constant with respect to data size so storing data for a long time is not a problem.

  3. Getting Started

    To get started you will essentially need to go through 4 steps:

    • Select region, organization and space for deployment of the services.
    • Install the IBM Cloud and IBM Functions CLI and login to update the components.
    • Provision the Weather Company Data Service.
    • Provision the IBM Event Streams service and create a topic for the service.

     

    Selection of deployment location

    IBM Cloud Functions is currently available in US South, US East, Germany and United Kingdom. The Weather Data Service is available in Dallas, London and Sidney. IBM Event Streams is available in the same regions as well as in Washington DC, Tokyo and Frankfurt. In this recipe we shall use London as the location for the Weather Company Data service and IBM Functions as well as  Germany (with its 3 availability zones in Frankfurt) as location for IBM Event Streams.

    Beyond the region you will also need to decide on which Cloud Foundry organization and space to use. Your default organization and the default dev space should do for a start, but feel free to make another choices.

     

    Installation of CLIs

    To prepare the development environment on your local machine, you will need to download and install the IBM Cloud Command Line Interface (CLI for short) and the IBM Functions CLI:

     

    When using the CLI you will need to login to a specific region. An important information is the one that defines the potential end-points for IBM Cloud (See ‘Regions‘ for more information):

    03.01-Login-Information

    We installed the tools on an Ubuntu Linux 64 bit virtual machine. To update the components (at any time) login using single sign on and follow the below procedure:

    1. Invoke the following command to set the endpoint and login
      ibmcloud api <ENDPOINT>
      ibmcloud login -sso
    2. This will display a link for obtaining a code for authentication.
      03.02-Login
    3. Enter Y and press Return.
    4. In the browser, position the cursor over the code and left-click the mouse to copy the one time code. The browser should respond with a popup message saying “Copied”.
    5. Paste the one time code into the command shell and click Return. You may not see the code even if it is correctly pasted into the shell.
    6. If the login procedure informs you that there is a new version of the CLI available, enter Y and press Return to accept the update.
    7. Check that the endpoint, organization and space are correctly set. Otherwise change the target:
      ibmcloud target -o <ORG> -s <SPACE>
      ibmcloud target --cf-api <ENDPOINT> -o <ORG> -s <SPACE>
    8. Use the following command to check the current installed version of IBM Cloud Functions and see if it is current:
      ibmcloud fn help
    9. Instrall or update the cloud functions plug in by executing one of the following commands
      ibmcloud plugin install cloud-functions
      ibmcloud plugin update cloud-functions

      03.03-Update-Functions

     It is a good idea to update the plugin at regular intervals to ensure that you work with the most recent version of IBM Cloud Functions CLI.

     

    Provisioning of the Weather Company Data service

    Next create an instance of the Weather Company Data service:

    1. Select the IBM Cloud Catalog again.
    2. Enter “weather” as search string.
    3. Select the ‘Weather Company Data’ service.
      03.04-Weather-Company-Data-service
    4. On the Weather Company Data page:
      1. Provide a name for the service (we optionally prefixed the service name with ‘Weather Data Demo’ but tags could be used as well).
      2. Select the region (e.g. Frankfurt).
      3. Choose an organization and a space (or accept the defaults).
      4. Add a tag (e.g. ‘weather-data-demo’) to the service. Tags will allow you to filter the resources later on so that you only see the services that are relevant for the current initiative.
      5. Scroll down and select for the pricing the Free Plan.
    5. Click Create to create the service.
    6. Click the New Credential button in the right part of the screen to create a new credential.
    7. Provide a name for the credential (e.g. ‘weather data demo credentials’).
    8. Click Add.
    9. For the new credential, select the View Credentials button in the Actions column.
    10. Copy all the information for the new credential to a file on your desktop. You will need it later on.

     

    Provisioning of the IBM Event Streams service

    Finally create an instance of the IBM Event Streams service. While doing so use the same values for the organization, space and tags as you used when creating the Weather Company Data service:

    1. Select the IBM Cloud Catalog again.
    2. Enter ‘Streams’ as search string.
    3. Select the ‘IBM Event Streams’ service.
      03.06-Event-Streams-Service
    4. On the Event Streams page:
      1. Provide a name for the service.
      2. Select the region (e.g. Frankfurt).
      3. Choose an organization and a space (or accept the defaults).
      4. Add a tag (e.g. ‘weather-data-demo’) to the service.
      5. Scroll down and select for the pricing the Standard Pricing Plan.
    5. Click Create to create the service.
    6. On the next page select the tab Service Credentials to the left.
    7. Click the New Credential button in the right part of the screen to create new a new credential.
    8. Provide a name for the credential (e.g. ‘weather data demo credentials’).
    9. Click Add.
    10. For the new credential, select the View Credentials button in the Actions column.
    11. Copy all the information for the new credential to a file on your desktop. You will need it later on.

     

    As a last step you will need to create a topic for the event stream that will be used to exchange weather data between producers and consumers:

    1. In the dashboard of the Event Stream service, select the Manage tab in the left toolbar.
      03.08-Create-Topic
    2. Click the + button to add a new topic.
    3. Enter ‘weather-data’ for the topic name.
      03.09-Topic-properties
    4. Keep the defaults for partition and retention period.
    5. Click Create Topic.

     

    To check the result select IBM Cloud > Resource List in the upper right corner of the IBM Cloud UI. Filter the elements using the ‘weather-data-demo’ tag and expand the section for Cloud Foundry Services:

    03.07-Tagged-Resources

  4. Weather Company Data Service and API

    The Weather Data Company provides a comprehensive set of APIs for retrieving past, current and forecasting weather data as well as weather data related to severe weather conditions, road conditions, agriculture, lightning‚Äôs and aviation. An overview of the offerings, together with a summary, usage guide and code examples is available on the webpage titled ‘IBM Passport Advantage Offerings | Weather Company Data‘. This page contains a link to the page ‘Weather Company Data | API | Common Usage Guide‘ which provides detailed information on how the URLs are composed as well as information regarding the returned status code when a REST request is made.

    To give you an expression on how the URLs may look like, here a few examples (with a geocode from the city of Munich):

    • Almanac (historic statistical data):
      https://api.weather.com/v3/wx/almanac/monthly/1month?geocode=48.1545703,11.2616557&format=json&units=h&month=12&apiKey=
    • 48-h forecast:
      https://api.weather.com/v3/wx/forecast/hourly/2day?geocode=48.1545703,11.2616557&format=json&units=m&language=de-DE&apiKey=
    • 5-Day daily forecast:
      https://api.weather.com/v3/wx/forecast/daily/5day?geocode=48.1545703,11.2616557&format=json&units=m&language=de-DE&apiKey=
    • 7-Day daily forecast:
      https://api.weather.com/v3/wx/forecast/daily/7day?geocode=48.1545703,11.2616557&format=json&units=m&language=de-DE&apiKey=
    • Current weather:
      https://api.weather.com/v3/wx/observations/current?geocode=48.1545703,11.2616557&units=m&language=de-DE&format=json&apiKey=

     

    The Weather Company Data service on IBM Cloud offers a limited set of capabilities as mentioned already in section 2. The REST API of that service in terms of the base URL¬† as well as the extensions for retrieving information related to historic, current and forecast weather conditions are documented on the web page titled ‘Weather Company Data For IBM Bluemix APIs‘.

    You can test the Weather Company Data service URLs e.g. by using curl or postman. For this purpose you will need the credentials that you created when you provisioned the service in the previous section. The curl command for a GET requests to receive a weather data observation is basically on the form:

    curl --user <user>:<password> --request GET 'https://twcservice.eu-gb.mybluemix.net/api/weather/v1/geocode/53.5453/9.9953/observations.json' | json_pp

    The URL consist of the base URL for the Weather data service (which includes the location ‘eu-gb’), the geocode and the designator for the kind of data being requested (‘observation.json’). The output delivered by the GET request is a flat string. By piping the output into json_pp we get a more readable output:

    04.01-Using-Curl-to-test-URL-1

    The returned data defines typical properties of the weather such as pressure, temperature, wind, snow and visibility as well as the time of the observation and the location in terms of name.

    Using postman, simply start the postman API, enter the URL, user name and password as shown below and click Send:

    04.02-Using-Postman-to-test-URL

    With Postman we can choose to view the output delivered by the service in its natural form as pretty printed JSON. In case you want to install and use Postman for retrieving Weather Data using REST calls:

     

    We have used the longitude 53.5453 and latitude 9.9953 which corresponds to a location in Hamburg. However, please feel free in the forthcoming sections of the recipe to use the geocodes for your location.  You can obtain the geocode using the location service of the Weather Company Data service. Here is an example how it can be achieved using the postal code as input :

    curl --user <user>:<password> --request GET https://twcservice.eu-gb.mybluemix.net/api/weather/v3/location/point?postalKey=30339:US&language=en-US | json_pp

    For more details please visit the online documentation for the Location Services of the Weather Company Data Service. Moreover, please feel free to use any geo location finder of your own choice, e.g.  the URL https://www.latlong.net/convert-address-to-lat-long.html.

  5. Setting up the Packages for IBM Functions

    In the forthcoming sections of the recipe you will create and configure IBM Cloud Functions so that it sends incoming weather data to IBM Event Streams. In IBM Cloud Functions use packages to bundle together a set of related actions, and share them with others. A package can include actions and feeds. An action is a piece of code that runs on Cloud Functions. For example, the IBM Event Streams package includes actions to write records to a topic of the IBM Event Streams. A feed is used to configure an external event source to fire trigger events. For example, the Alarm package includes a feed that can fire a trigger at a specified frequency. For more information we refer to the section of online documentation titled ‘Organizing actions in packages‘.

    The solution for reading weather data and sending that data to IBM Event Streams requires the use of 3 packages:

    1. A package for the Weather Company Data service (available but needs to be bound to the credentials for the service).
    2. A package for the IBM Event Streams service (available but needs to be bound to the credentials for the service).
    3. A package for the application specific logic.

     

    IBM Functions has pre-installed packages are automatically registered within Cloud Functions in the /whisk.system namespace. Although it is possible to use the entities in a package directly, there may be situations where you wpould need to pass the same parameters to the action every time. You can simplify the process by binding to a package and specifying default parameters, which are inherited by the actions in the package. Such default parameters could for example define the credentials of the service. To store credentials or other parameters in a pre-installed package, you must consequently create package bindings.

    The application specific package will need one trigger, one sequence and four actions as shown below:

    01.1-Functions-Architecture

    The sequence is a chain of actions where output of one action acts as an input to the following action and so on.  This allows you to combine existing actions together for quick and easy re-use. The input to and output from an action is a dictionary of key-value pairs. The key is a string, and the value a valid JSON value.

    A sequence can then be invoked just like an action, through a REST API, manually using the user interface or automatically in response to events.  Such an event may be a trigger configured to fire at specific time intervals so that weather data are ingested periodically Рbe it every 5th minute (during development and test) or every hour (for production purposes). 

    The sequence itself has four actions:

    • prepareParams that defines the location (latitude, longitude) and observation time (current) for the weather data to be ingested.
    • forecast that actually retrieves the weather data from the Weather Company Data service.
    • prepareMessage that post-processes the JSON object retrieved from the Weather Company Data service before it is passed to IBM Event Streams.
    • messageHubProduce that writes the JSON object to a specific topic of the IBM Event Streams service.

     

    You provide your action to Cloud Functions either as source code or as a Docker image. In this recipe we shall keep it simple and develop small minimal fragments of Node.js code. When creating actions it is important to notice that action implementations are stateless, or idempotent. While the system does not enforce this property, it is not guaranteed that any state maintained by an action is available across invocations. Moreover,¬† Cloud Functions has a few system limits, including how much memory an action can use and how many action invocations are allowed per minute. For more detail on these limits please visit ‘System Limits‘.

    Beyond creating the packages, the sequence, the trigger and relevant actions, there will be additional activities involved related to checking and monitoring the results produced by IBM Cloud Functions, which will get you to use the command line interface, the code editor and the monitoring dashboard.

     

    Setting up the Weather Data Company package

    To bind the package for the Weather Company Data service you will need the command line interface as well as the credentials for the service:

    1. Set the endpoint for IBM Cloud to the designated region in a command shell as shown in section 3 (Getting started)
    2. Log into IBM Cloud using SSO as shown in section 3.
    3. Obtain the user name, password and host name for the Weather Company Data service as shown in section 3.
    4. In the CLI, submit the following command using your values for username, password and hostname.
      ibmcloud fn package bind /whisk.system/weather weather-data-company -p username "<username>" -p password "<password>" -p host "twcservice.eu-gb.mybluemix.net"

     

    The following image shows the commands:

    05.02-CLI-Package-Setup

    You can immediately test if the installation of the package has worked according to intention by submitting the following command to retrieve the weather forecast:

    ibmcloud fn action invoke weather-data-company/forecast --result --param latitude 43.7 --param longitude -79.4

    You can find more information on the Weather Company Data package on the section “Weather package” in the online documentation.

     

    Setting up the Kafka Package

     Next, bind the Kafka package. You will again need the command line interface as well as the credentials for the Kafka service:

    1. Obtain the kafka_brokers_sasl, user name, password and kafka admin URL from the IBM Event Stream service as shown in section 3.
    2. Change the kafka_brokers-sasl property so that it becomes a single line string with quotation marks properly escaped using backslash as shown below.
    3. In the CLI, submit the following command using your values for kafka_brokers_sasl, user, password and kafka_admin_url:

    ibmcloud fn package bind /whisk.system/messaging weather-data-event-streams -p kafka_brokers_sasl "[\"kafka04-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka02-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka01-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka03-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka05-prod01.messagehub.services.eu-de.bluemix.net:9093\"]" -p user "<username>" -p password "<password>" -p kafka_admin_url "https://kafka-admin-prod01.messagehub.services.eu-de.bluemix.net:443"

     

    You may find the following CLI commands useful when working with IBM Functions:

    • To get help: ibmcloud fn help
    • Show the installed packages: ibmcloud fn package list
    • Update the installed packages: ibmcloud fn package refresh
    • Retrieve list of actions: ibmcloud fn action list

     

    If you would like to change the target IBM Event Stream service at some point in time so that the weather observations are send to another instance of the server, then it is easily done using the package update command, but be aware that you will need to define bindings for each single parameter (just updating e.g. the user name and the password will not do):

    ibmcloud fn package update weather-data-event-streams -p kafka_brokers_sasl "[\"kafka04-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka02-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka01-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka03-prod01.messagehub.services.eu-de.bluemix.net:9093\",\"kafka05-prod01.messagehub.services.eu-de.bluemix.net:9093\"]" -p user "<username>" -p password "<password>" -p kafka_admin_url "https://kafka-admin-prod01.messagehub.services.eu-de.bluemix.net:443"

    Notice that the command prefix ibmcloud fn is equivalent to the original OpenWhisk command wsk.

     

    Viewing Packages in IBM Cloud console

    Having created the package bindingswe can check if the packages have been created correctly using the IBM Cloud dashboard. To achieve this do the following:

    1. In the IBM Cloud portal select the Catalog.
    2. Enter ‘Function’ as filter and locate the icon for IBM Functions.
    3. Click the Functions service to open it.
    4. Click Actions in the toolbar to the left of the page.
    5. Select the correct region, Cloud Foundry organization and space in the toolbar at the top of the page.
    6. Scroll down and expand the package bindings named ‘weather-data-event-streams’ and ‘weather-data-company’:
      05.09-Packages
    7. Click the forecast action and wait until the code is loaded:
      05.10-Forecast-action
       

    Notice that the action uses parameters such as those defining the credentials for the Weather Company Data service. These credentials come from the package binding which has the advantage that secrets aren’t hardcoded into code. However, there are more parameters than just the secrets. The switch statement needs a time period for the forecast which can take values such as 48hours, current or timeseries. Moreover, we need to specific the location in terms of latitude and longitude for the measurement. These parameters can be specified by defining an action that is executed before the forecast function is called, thus passing the values of those missing parameters to the forecast function.

     

  6. Ingestion of Weather Data using IBM Functions

    With the predefined packages correctly configured in IBM Functions we can continue ingesting weather data using IBM Functions.  This will require several steps

    1. Create a package for the current application.
    2. Create sequence that chain together the various actions needed.
    3. Create an action to read weather data from the Weather Company Data service.
    4. Create an action that defines the location of the weather data to be retrieved.
    5. Test the sequence.

     

    Create initial package, sequence and action for the current application

    Having created the package bindings we will continue creating a package for the application as well as an initial sequence that contains an action for reading weather company data. This corresponds to the steps 1-3 mentioned in the beginning of this section.

    To achieve this do the following:

    1. In the IBM Cloud portal, select Getting Started > Overview in the toolbar to the left:
      05.03-IBM-Cloud-Functions
    2. Click the Start Creating button.
    3. In the upper part of the screen select the appropriate Region, Cloud Foundry Org and Cloud Foundry Space:
      05.04-Create-Sequence
    4. Select Create Sequence.
    5. On the next screen enter ‘observeWeather’ as the name of the sequence.
    6. Right to the field Enclosing Package, click Create Package to create a new package.
    7. Enter ‘weather-data-demo’ as the package name and click Create.
    8. For the action, select Use Public to start the sequence with a weather data action.
      05.05-Create-Sequence-Public-Action
    9. Select the package named Weather.
    10. In the Actions field, select the ‘forecast’ action.
    11. For the binding to a package, select My Bindings.
      05.06-Completed-Sequence-Configuration
    12. Select the package ‘weather company data’ that you just created using the CLI
    13. Click Create.

      05.07-Initial-Sequence

     The initial sequence contains a single action to fetch weather data from the Weather Company Data service that you provisioned in section 3.

     

    Create action defining the weather data location and time period

    To be able to test the sequence it will be necessary to create a function that will pass latitude, longitude and time parameter values to the forecast function. This is achieved in Node.js by the following function definition:

    function main(params) {
        return {
           latitude: 53.5453,
           longitude: 9.9953,
           timePeriod: 'current'
        };
    }

    To create and test the action (as well as the sequence) in IBM Cloud do the following:

    1. Navigate to the page showing the ‘observeWeather’ sequence.
      05.11-Add-Action
    2. Click the Add button in the list of Sequence Actions.
    3. On the next page name the action ‘prepareParams’.
      05.12-Action-Configuration
    4. Select ‘weather-data-demo’ as the package for the action.
    5. Select ‘Node.js 10’ for the runtime.
    6. Click Create & Add.
    7. This will create the action and the sequence will now have two actions, but’ createParams’ will be the second.
      05.13-Action-Order
    8. Click the Up-Arrow for the ‘createParams’ action to move it one step up in the list in front of the forecast action.
    9. Click the link named ‘createParams’ to open the code editor for the action.
    10. Change the code body to the following:
      05.14-PrepareParams-Code
    11. Click Save to save the changes.
    12. Click Invoke to test the function.
      05.15-PreparaParams-Invocation
    13. Click the Collapse button in the top part of the screen to collapse the view.
    14. Click the browsers Back Arrow to go to the previous page (which shows the sequence).
    15. Click Save to save the changes to the sequence.

     

    Test the sequence

    To test that reading data from the Weather Data Company service works, do the following:

    1. Click Invoke.
    2. Wait for the result to appear:
      05.16-Weather-Observation-Data
    3. IBM Functions will return the result as well as the time used for the computation.

     

    You have now ingested weather data to IBM Cloud using IBM Functions. Notice that the result is of type observation and that the data returned cover a variety of properties relevant for describing the current weather condition in a given location (in this case Hamburg Finkenwerder). Also observe that the data is a JSON value that contains meta data as well as the observation data.

  7. Sending Weather Data to IBM Event Streams using IBM Functions

    Having read the weather data only a few steps are needed in order to write the data to IBM Event Streams:

    1. Create the action ‘prepareMessage’ that prepares the weather data for delegation to IBM Event Streams.
    2. Create the action that sends the data to IBM Event Streams.

     

    Create action that prepares the weather data for IBM Event Streams

    The weather data received from the Weather Company Data service can’t be forwarded directly to IBM Event Streams since it takes the form of a JSON object and IBM Event Streams expects a flat string value. The JSON object must therefore be converted to a string value. Moreover, if we look that the data received in the screenshot above, then it can be observed that the location parameters are part of the metadata and not the message itself. We would consequently like to add the latitude and longitude properties to the message before it is passed onwards to Kafka. Basically this means that we will need to define an action that has the following function definition:

    function main(params) {
        params.observation.latitude = params.metadata.latitude;
        params.observation.longitude = params.metadata.longitude;
        return {
           topic: "weather-data",
           value: JSON.stringify(params.observation)
        };
    }

    Define a new action following the same steps as you just did to define the prepareParams action:

    1. Go to the page that defines the ‘observeWeather’ sequence.
    2. Click Create New to create a new action.
    3. Name the new action ‘prepareMessage’ and use the following settings to define the action.
      05.17-PrepareMessage-Action
    4. Select the new action in the sequence (it should be the last in the list).
      05.18-Sequence-with-PrepareMessage
    5. Define the semantics of the action as shown below.
      05.19-PrepareMessage-Code
    6. Save the new sequence.
    7. Test the sequence by clicking the Invoke button.
      05.20-PrepareMesssage-Results

     

    Create action that sends the message to IBM Event Streams

    The final action needed in the sequence is to send the message to IBM Event Streams. To achieve this do the following:

    1. Create a new action for the sequence.
    2. In the Add Action dialog, select Use Public.
    3. Select the MessageHub package.
      05.21-SendMessage-Action
    4. Select the ‘messageHubProduce’ action.
    5. Select the binding My Bindings.
    6. Select the binding named ‘weather-data-event-streams’.
    7. Click Add to create the new action.
    8. The new action should now be added at the end.
      05.22-Final-Sequence
    9. Save the sequence.
    10. Invoke the sequence.

     

    You should get the message in the log that the weather data has been send successfully to the Event Stream service.

  8. Scheduling of IBM Functions using Triggers

    The defined sequence has been executed manually so far. A better approach would be to schedule it so that data are fetched from the Weather Company Data service at regular intervals. This can be achieved by defining an associated trigger for the sequence.

    During testing it would make sense to trigger the sequence every minute (or every fifth minute depending on your preferences and patience). However, for a production system every hour would probably be a more relevant choice.

    To create the trigger do the following:

    1. On the page that defines the sequence, select connected triggers in the toolbar to the left.
    2. Click the Add Trigger command in the top right part of the page.
    3. On the next page select Periodic as trigger type.
      05.26-Trigger-Name
    4. Name the trigger ‘observeWeatherTrigger5Min’.
    5. Scroll down to the control that is named UTC Minutes and set the pattern to ‘Every 5 Minutes’.
    6. Click Create and Connect.
    7. Check that the trigger has now been created and that it is enabled.
      05.27-Connected-Triggers

     

    Do not forget to change the trigger so that it only triggers the sequence every hour (or never – depending on your preferences) before you finish work on this recipe. Remember: running systems in Cloud may incur costs. Alternatively, disable the trigger by deselecting the check-button for the trigger.

  9. Monitoring of IBM Functions

    All logs and activation records are automatically available via IBM Cloud Log Analysis Service which provides rich Kibana-based experience for searching logs in any dimension. This feature includes visualization capabilities which are critical for quickly detecting trends and anomalies and for problem determination at scale.

    To view the log do the following:

    1. Select the sequence that you defined.
    2. Click Log in the toolbar to the left
    3. Once on the window for IBM Cloud Logging, double click Discover in the toolbar to the left.
      09.02-Log-Monitoring

    You can get insight into the performance of your actions that are deployed with Cloud Functions by using IBM Cloud Monitoring. You can also monitor the health and performance of your actions by using the dashboard to see a graphical summary of your activity.

    An overview of the activity summary, the activity log entries and the timeline do the following:

    1. Select Monitor in the toolbar to the left.
    2. View the summary of the activities for the sequence that you defined:
      09.01-Monitoring
    3. Notice that you can filter the view to display specific actions.
    4. Using the activity log you can easily get a detailed view of the log messages.
  10. Receiving Weather Data from IBM Event Streams using IBM Functions

    Apache Kafka offers several kind of API’s that can be used by Apache Kafka clients – be it Java, Scala, Node.js or Python clients (to mention a few):

    • A Producer API that allows an application to publish records to one or more Kafka topics.
    • A Consumer API that allows an application to subscribe to one or more topics and process the records of the topics.
    • A Streams API allows an application to act as a stream processor, consuming an input stream from one or more topics and producing an output stream to one or more output topics, effectively transforming the input streams to output streams.
       

    Using IBM Functions you can consume IBM Event Stream messages from a specific topic. This simply requires the definition of a trigger that will then fire whenever a message comes available in the topic. With the trigger as a starting point it is then possible to further process the message using actions and to chain actions using sequences.

    We shall therefore just show the creation of the trigger which can be created as follows:

    1. In the IBM Cloud portal for functions select, select Triggers in the toolbar to the left.
    2. On the next page titled ‘Triggers’, click the Create button.
    3. On the next page titled ‘Create’, select Create Trigger.
    4. On the next page titled ‘Connect Trigger’ select MessageHub.
    5. On the next page:
      10.01-Create-Trigger
      1. Provide a name for the trigger (e.g. ‘consumeWeatherData’).
      2. Select the MessageHub instance.
      3. Select the topic to subscribe to (‘weather-data’).
      4. Select Is JSON Data.
    6. Click Create to create the trigger

     

    Once the trigger is created you can combine it with actions to process the data read as has already been demonstrated in the previous sections.  You can achieve the same using the CLI. To create a trigger for an IBM Event Stream topic, submit a command on the following form replacing the placeholders with your service and topic name accordingly:

    ibmcloud fn trigger create MyMessageHubTrigger -f <myMessageHub>/messageHubFeed -p topic <mytopic> -p isJSONData true

  11. On Demand Ingestion of Weather Data

    So far we have worked with current weather data represented in form of fairly small JSON files that were ingested to the IBM Cloud and delegated onwards to IBM Event Streams on a timely basis using triggers. However, we can also use the Weather Data API to get information about for example lightning’s World Wide, and in that case we may be dealing with information around the size of several MBs. Of course we could ingest that information at regular intervals as well (e.g. every minute) but that would only make sense if there was a business need for doing so. There may be situations where we would rather like to invoke the ingestion from the outside Рon demand Рusing a REST call rather than doing it automatically and periodically. In this section we shall show you how to achieve this using web actions. Rather than using the Weather Data API which would require you to obtain an API Key, we will show how to trigger the ingestion using an URL that can be served by the Weather Company Data service on IBM Cloud. In principle however, the solution applies equally well to the Weather Company API. It is just the URL that is different.

    If you navigate to your Weather Company Data service in IBM Cloud and view the service credentials for the service, you will see that the credentials provide 5 properties, namely the user name, the password, the host, the port and the URL. The URL contains the user name and the password as well:

    11.01-Service-Credentials

    In this section we shall use the URL on the form above since it is more similar to the Weather Company API where the API Key is given directly in the URL. We shall furthermore use an URL that retrieves daily weather forecast information:

    https://twcservice.eu-gb.mybluemix.net/api/weather/v1/geocode/53.5453/9.9953/forecast/daily/3day.json

    Notice that the service accepts 3, 5, 7 and 10 days forecasts. Another relevant URL to try out would be the one that delivers intraday forecast in segments of 6 hour every day for either 3, 5, 7 or 10 days:

    https://twcservice.eu-gb.mybluemix.net/api/weather/v1/geocode/53.5453/9.9953/forecast/intraday/10day.json

    The IBM Functions sequence will be invoked by an external REST call (a web action).¬† The first steps in the sequence named ‘prepareURL’ will return as parameter the URL to be used to fetch the weather data – be it from the Weather Company API or from the Weather Company Data service on IBM Cloud.¬† This URL will then be passed onwards to the second step which will use the curl utility action of IBM Functions to submit a REST call that retrieves the weather data from the corresponding service:

    11.02-Sequence-Architecture

    The last two steps are the same as in the sequence defined previously. We will therefore focus on creating the sequence, its trigger and the first two actions. All in all it will require the following steps:

    1. Create sequence.
    2. Add the initial curl function of the samples package to the sequence.
    3. Add the ‘prepareURL’ action as first action to the sequence with code that returns the URL to be used by the curl call.
    4. Test the sequence by invoking it.
    5. Create a web action for the sequence so that the sequence can be invoked by a URL.
    6. Test the URL.

     

    Go the IBM Functions and make sure that you are in the correct region. To create the sequence do the following:

    1. Select Actions and invoke  the Create Action command.
    2. Select Sequence.
    3. Name the sequence ‘getForecast’, select ‘weatherdatademo’ as package and ‘Use Public’ for the action to be inserted.¬†
    4. Select Samples as the public package to use.
    5. Select curl as the action of the sequence.
      11.03-Create-Squence
    6. Click Create to create the sequence with the first action contained.

     

    Once the sequence is created, open the curl action and observe the code:

    11.04-Curl-Code

    The action requires as parameter the URL to be used for the REST request. This means that for a quick solution you can simply insert an action ‘prepareURL’ that will return a message with a payload parameter that contains the Weather Data URL.

    The sequence should look as follows:

    11.07-Sequence-with-Actions

    The code for the paramURL action should be as follows (using the credentials for your instance of the Weather Company Data service):

    function main(params) {
        return {
            payload : 'https://<username>:<password>@twcservice.eu-gb.mybluemix.net/api/weather/v1/geocode/53.5453/9.9953/forecast/daily/3day.json'
        };
    }

    Notice that all it needs to read data from the Weather Company API rather than the service on IBM Cloud would be to change the URL.

    Having completed the sequence you can test the work by clicking the Invoke button for the sequence. This should return a JSON object with the relevant information for the URL. What is missing is to define a web action for the sequence so that the invocation can be triggered by a REST request, which requires the following steps:

    1. Select the sequence named ‘getForecast’.
    2. Select the Web Action button to the right of the sequence name:
      11.09-Web-Action
    3. Select Enable as Web Action.
    4. Click the Copy icon to the right of the URL of the web action to get the URL for the new web action.

     

    To test the URL (in our case ‘https://eu-de.functions.cloud.ibm.com/api/v1/namespaces/einar.karlsen%40de.ibm.com_dev/actions/weatherdemo/getForecast’) simply paste it into a web browser and wait for the result to occur.¬†

    It should be noted that the current solution using Web Actions would allow anyone to initiate the request. This can be avoided e.g. by using API Keys or by using securing the Web Action as described in the online documentation ‘Creating Web Actions‘. If you click the API-KEY link above you can obtain the API Key for your name space:

    11.10-API-Key

     We shall leave the discussion regarding protection of the web actions as it is for now since it goes beyond the scope of the current recipe.

  12. Conclusion

    In this recipe we have shown how to ingest weather data coming from the Weather Company using IBM Cloud services covering 2 key steps: ingestion of weather data into the IBM Cloud using the Weather Company API and IBM Cloud Functions, and delegation of the captured weather data for further processing to IBM Event Streams. For this purpose we have used key features of IBM Functions (and thus Apache OpenWhisk) such as packages, sequences, actions, triggers and web actions. With IBM Functions we have shown how to produce messages for IBM Event Streams as well as how to consume messages and how to test and monitor that the operations work correctly.

  13. Acknowledgement

    This recipe represents joint work between Julia Wiegel, Rene Meyer and Einar Karlsen. We would like to thank Paul Smith from the Weather Company for providing input regarding the Weather Company API and use of Node.js clients to ingest weather data using the API.

Join The Discussion