Overview

Skill Level: Intermediate

Familiarity with Lisp, Scheme or similar languages is a plus.

Common Lisp is a very special language with an unmatched expressive power; with Armed Bear Common Lisp we can leverage existing Java libraries and thus bring that expressiveness to situations where we would otherwise be limited in our language choice.

Ingredients

Step-by-step

  1. Deploy an instance of the Watson IoT Platform

    Follow the Getting Started instructions from the official documentation to create a Watson IoT Platform instance using you Bluemix account.

    1. Go to the Watson IoT Platform catalog entry
    2. Accept the provided name or change it to something to your liking
    3. Click Create

    You should now see the Watson IoT  welcome page

    welcome_watson_iot

  2. Create the IoT device

    We will use Common Lisp to send events to Watson IoT; how we get the data for those events is not very important, we could generate random values for example (and indeed that’s what we will end up doing)

    To receive data we will create an IoT device under a custom device type (which we will also create)

    watson_devices

     

    Select Add Device... in the Devices pane, and Create device type; give it a name and accept the default values to create the device type

     

    create_dev_type

     

    Now we can create a new device of the type we created (lispything, in our example) by selecting it in the Choose Device Type dropdown. The Device ID is the only required information and we can use any value – cl1 for example.

    Going forward we will be asked for token information in the Security pane; the token “… must be between 8 and 36 characters long, and should contain a mix of lower and upper case letters, numbers, and symbols (hyphen, underscore, exclamation-point, ampersand, at sign, question mark, period, right and left parentheses are permitted)…” according to the documention, although given the scope of this recipe it doesn’t really matter – I’ll use 01234567 . Whatever you end up using be sure to save it somewhere.

    The device is created and a summary of the configuration presented.

    dev_created

  3. Test the device via REST API

    Using the REST API we will make a first test; the Org ID is displayed near the username and is needed to interact with Watson IoT. The REST API documentation provides the details

    $ curl -u "use-token-auth:<AUTH_TOKEN>" -H Content-Type:application/json --data-ascii "{\"temperature\":20}" https://<ORG_ID>.messaging.internetofthings.ibmcloud.com:8883/api/v0002/device/types/<DEV_TYPE>/devices/<DEV_ID>/events/<EVENT>

    Replacing with adequate values considering our steps thus far:

    $ curl -u "use-token-auth:01234567" -H Content-Type:application/json --data-ascii "{\"temperature\":20}" https://72o9h9.messaging.internetofthings.ibmcloud.com:8883/api/v0002/device/types/lispything/devices/cl1/events/temp

    The device should, at the time it receives the messages, be listed as “Connected” as indicated by the icon just before the device id

    connected

    Creation of Watson IoT board cards is outside the scope of this recipe but it’s simple to go to Boards > Device-Centric Analytics > +Add New Card and create a new Line Chart card which tracks the temp value

    line_chart_create

    Changing the values in the submitted JSON (via curl, as above) produces something like this:

    line_chart_view

    The IoT device is connected and the REST API works.

  4. Setting the Common Lisp environment

    Download ABCL from the Armed Bear Common Lisp site; the archive contains amongst others two JAR files – abcl.jar and abcl-contrib.jar – which are the essential components of ABCL’s implementation of Common Lisp on the JVM.

    ABCL is a conforming implementation of ANSI Common Lisp that runs in a Java virtual machine and compiles Lisp code directly to Java byte code; at the most basic this means that we can use ABCL like we would use any other Common Lisp compiler (and indeed that’s what I mostly do, even when not taking advantage of the Java interop functionality) with the advantage of being available anywhere where a JVM is and, perhaps more importantly, providing specific integration abilities to use Java libraries and code inside Lisp.

    NB: since ABCL follows ANSI Common Lisp we could – and should – use the typical development tooling which Common Lisp projects adopt, like Quicklisp, Quickproject, ASDF, etc. . This recipe will not provide much explanation on them since it assumes either a knowledge of Common Lisp of the willigness to explore these matters separately. This also applies to Emacs and the Superior Lisp Interaction Mode.

    The Lisp REPL (Read-eval-print loop) is obtained by simply adding the JAR files to the classpath and calling  org.armedbear.lisp.Main:

    $java -server -Xrs -cp /opt/abcl-bin-1.4.0/abcl.jar:/opt/abcl-bin-1.4.0/abcl-contrib.jar: org.armedbear.lisp.Main

    I use the following script to simplify the process and simply use abcl as the command

    #!/bin/sh
    ABCL_JAR=/opt/abcl-bin-1.4.0/abcl.jar
    ABCL_CONTRIB=/opt/abcl-bin-1.4.0/abcl-contrib.jar
    JAVA=$(which java)
    ABCL="$JAVA -server -Xrs -cp $ABCL_JAR:$ABCL_CONTRIB:$JNA org.armedbear.lisp.Main"
    exec $ABCL "$@

    Running abcl results in a standard Lisp REPL, with some additional information about the JRE environment being used:

    $ abcl
    Armed Bear Common Lisp 1.4.0
    Java 1.7.0 IBM Corporation
    IBM J9 VM
    Low-level initialization completed in 0.398 seconds.
    Startup completed in 1.715 seconds.
    Using probed value of abcl-contrib:
    '/opt/abcl-bin-1.4.0/abcl-contrib.jar'.
    Added jar:file:/opt/abcl-bin-1.4.0/abcl-contrib.jar!/quicklisp/ to ASDF.
    Added jar:file:/opt/abcl-bin-1.4.0/abcl-contrib.jar!/mvn/ to ASDF.
    Added jar:file:/opt/abcl-bin-1.4.0/abcl-contrib.jar!/jss/ to ASDF.
    Added jar:file:/opt/abcl-bin-1.4.0/abcl-contrib.jar!/jfli/ to ASDF.
    Added jar:file:/opt/abcl-bin-1.4.0/abcl-contrib.jar!/asdf-jar/ to ASDF.
    Added jar:file:/opt/abcl-bin-1.4.0/abcl-contrib.jar!/abcl-asdf/ to ASDF.
    Loading /home/fsmunoz/.abclrc completed in 9.06 seconds.
    Type ":help" for a list of available commands.
    CL-USER(1): (list 1 "two" 3)
    (1 "two" 3)

     

  5. Creating our Common Lisp project

    We will use Quickproject to speed up things; this article from Zach Beane, author of Quicklisp and Quickproject, explains the process in more detail, including the installation of Quicklisp itself.

    CL-USER(10): (quickproject:make-project "~/src/dW/cl-watson-iot")
    WARNING: Coercing "~/src/dW/cl-watson-iot" to directory
    "cl-watson-iot

    Quickproject created the project directory and the initial files

    $ ls -l cl-watson-iot/
    total 16
    -rw-rw-r--. 1 fsmunoz fsmunoz 361 Apr 22 15:30 cl-watson-iot.asd
    -rw-rw-r--. 1 fsmunoz fsmunoz 110 Apr 22 15:30 cl-watson-iot.lisp
    -rw-rw-r--. 1 fsmunoz fsmunoz 63 Apr 22 15:30 package.lisp
    -rw-rw-r--. 1 fsmunoz fsmunoz 61 Apr 22 15:30 README.tx

    Using Emacs with SLIME we can interactively define and evaluate functions, like this hello one:

    ;;;; cl-watson-iot.lisp

    (in-package #:cl-watson-iot)

    ;;; "cl-watson-iot" goes here. Hacks and glory await!

    (defun hello ()
    (print "Hello developerWorks!"))

    To reflect the changes we should now reload the system (which is automatically created by make-project).

    CL-USER(4): (ql:quickload "cl-watson-iot")
    To load "cl-watson-iot":
    Load 1 ASDF system:
    cl-watson-iot
    ; Loading "cl-watson-iot"

    ("cl-watson-iot")

    The hello  function is callable by using the package name

    (cl-watson-iot::hello)

    The result should be something similar to the following

    hello_dw

    With the project set up and our environment working it’s time to interact with Watson IoT from Lisp.

  6. Watson IoT from Lisp, using the REST API

    Our very first step will be to use the same approach as before (using the REST API) but in Common Lisp; for that we will use the Drakma library. Since we are using Quicklisp this is a matter of adding a dependency on the cl-watson-iot.asd file

    (asdf:defsystem #:cl-watson-iot
    :description "Describe cl-watson-iot here"
    :author "Your Name <your.name@example.com>"
    :license "Specify license here"
    :defsystem-depends-on (:abcl-asdf)
    :depends-on (#:drakma)
    :serial t
    :components ((:file "package")
    (:file "cl-watson-iot")))

     Quickloading the system again will pick up the changes and retrieve the necessary dependencies:

    CL-USER(2): (ql:quickload "cl-watson-iot")
    To load "cl-watson-iot":
    Load 1 ASDF system:
    cl-watson-iot
    ; Loading "cl-watson-iot"
    ................
    ARTIFACT_RESOLVING net.java.dev.jna:jna:pom:4.2.2
    ARTIFACT_DOWNLOADING net.java.dev.jna:jna:pom:4.2.2 @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ARTIFACT_DOWNLOADED net.java.dev.jna:jna:pom:4.2.2 (/home/fsmunoz/.m2/repository/net/java/dev/jna/jna/4.2.2/jna-4.2.2.pom) @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ARTIFACT_RESOLVED net.java.dev.jna:jna:pom:4.2.2 (/home/fsmunoz/.m2/repository/net/java/dev/jna/jna/4.2.2/jna-4.2.2.pom) @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ARTIFACT_RESOLVING net.java.dev.jna:jna:jar:4.2.2
    ARTIFACT_DOWNLOADING net.java.dev.jna:jna:jar:4.2.2 @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ARTIFACT_DOWNLOADED net.java.dev.jna:jna:jar:4.2.2 (/home/fsmunoz/.m2/repository/net/java/dev/jna/jna/4.2.2/jna-4.2.2.jar) @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ARTIFACT_RESOLVED net.java.dev.jna:jna:jar:4.2.2 (/home/fsmunoz/.m2/repository/net/java/dev/jna/jna/4.2.2/jna-4.2.2.jar) @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ..................................
    [package cffi]....................................
    [package cffi-features]...........................
    [package trivial-garbage].........................
    [package cl+ssl]..................................
    [package drakma]..................................
    [package cl-watson-iot]; Compilation unit finished
    ; The following functions were used but not defined:
    ; CL+SSL::SSL-CTX-SET-DEFAULT-VERIFY-DIR
    ; CL+SSL::SSL-CTX-SET-DEFAULT-VERIFY-FILE


    ("cl-watson-iot")

     We can see an example of ABCL integration with the Java world: abcl-asdf adds the ability to use Maven dependencies with ASDF (the system definition facility used in most Common Lisp projects), something which is done indirectly in this case but which we will use explicitly latter on.

    The send-data function uses Drakma to send a POST request with Basic Authentication and a JSON payload, similar to our previous example with curl.

    ;; Some global variables to simplify 
    (defvar *org-id* "72o9h9" "Org ID")
    (setq *uri* (format nil "https://~A.messaging.internetofthings.ibmcloud.com:8883/api/v0002" *org-id*))
    (defvar *device-type* "lispything")

    (defun send-data (device event property value &optional (uri *uri*) (device-type *device-type*))
    "Sends VALUE associated with KEY to DEVICE of DEVICE-TYPE in URI endpoint"
    (drakma::http-request
    (format nil "~A/device/types/~A/devices/~A/events/temp" uri device-type device event)
    :basic-authorization '("use-token-auth" "01234567")
    :content-type "application/json"
    :method :post
    :content (format nil "{\"~A\":~A}" property value)))

    Using the appropriate parameters we can send data to our device:

    (send-data "cl1" "temp" "temperature" 16)
    ...

    POST /api/v0002/device/types/lispything/devices/cl1/events/temp HTTP/1.1
    Host: 72o9h9.messaging.internetofthings.ibmcloud.com:8883
    User-Agent: Drakma/2.0.2 (Armed Bear Common Lisp 1.4.0; Linux; 2.6.32-642.15.1.el6.x86_64; http://weitz.de/drakma/)
    Authorization: Basic dXNlLXRva2VuLWF1dGg6MDEyMzQ1Njc=
    Accept: */*
    Connection: close
    Content-Type: application/json
    Content-Length: 18

    HTTP/1.1 200 OK
    Server: IBM IoT MessageSight
    Date: Sat, 22 Apr 2017 23:13:49 GMT
    Access-Control-Allow-Origin: *
    Access-Control-Allow-Credentials: true
    Connection: close
    Keep-Alive: timeout=60
    Cache-Control: no-cache
    Content-Type: text/plain;charset=utf-8
    Content-Length: 0

    With this function we can already use other Lisp constructs, of course; let’s try the infamous loop:

    (loop for x from 10 to 40 do
    (send-data "cl1" "temp" "temperature" x))

    This loop sends a quick succession of values which are charted as close data points in the Watson IoT card:

    loop_chart

     

  7. Using Eclipse Paho MQTT client

    We have used the REST API for testing but we want to use MQTT; as the documentation says

    MQTT is the primary protocol that devices and applications use to communicate with the IBM Watson‚ĄĘ IoT Platform. Client libraries, information, and samples are provided to help you to connect and integrate your devices with Watson IoT Platform

    One of the most popular MQTT client implementations is Eclipse Paho, which has an MQTT client implementation in Java. We will make use of ABCL’s interoperability capabilities to use Eclipse Paho functions in a seamless manner.

    NB: There is also a Watson IoT specific Java interface which we could use; the same principles showned here are easilly applicable to that scenario as well.

    The first step is to declare the dependency by adding it to the ASDF file; the :mvn keywork specifies a Maven dependency.

    ;;;; cl-watson-iot.asd

    (asdf:defsystem #:cl-watson-iot
    :description "Describe cl-watson-iot here"
    :author "Your Name <your.name@example.com>"
    :license "Specify license here"
    :defsystem-depends-on (:abcl-asdf)
    :depends-on (#:drakma)
    :serial t
    :components ((:mvn "org.eclipse.paho/org.eclipse.paho.client.mqttv3/1.1.1")
    (:file "package")
    (:file "cl-watson-iot")))

    Doing a (ql:quickload "cl-watson-iot") will retrieve the JAR file from the Maven central repository

    CL-USER(7): (ql:quickload "cl-watson-iot")
    To load "cl-watson-iot":
    Load 1 ASDF system:
    cl-watson-iot
    ; Loading "cl-watson-iot"
    ARTIFACT_RESOLVING org.eclipse.paho:org.eclipse.paho.client.mqttv3:pom:1.1.1
    ARTIFACT_RESOLVED org.eclipse.paho:org.eclipse.paho.client.mqttv3:pom:1.1.1 (/home/fsmunoz/.m2/repository/org/eclipse/paho/org.eclipse.paho.client.mqttv3/1.1.1/org.eclipse.paho.client.mqttv3-1.1.1.pom) @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ARTIFACT_RESOLVING org.eclipse.paho:java-parent:pom:1.1.1
    ARTIFACT_RESOLVED org.eclipse.paho:java-parent:pom:1.1.1 (/home/fsmunoz/.m2/repository/org/eclipse/paho/java-parent/1.1.1/java-parent-1.1.1.pom) @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    ARTIFACT_RESOLVING org.eclipse.paho:org.eclipse.paho.client.mqttv3:jar:1.1.1
    ARTIFACT_RESOLVED org.eclipse.paho:org.eclipse.paho.client.mqttv3:jar:1.1.1 (/home/fsmunoz/.m2/repository/org/eclipse/paho/org.eclipse.paho.client.mqttv3/1.1.1/org.eclipse.paho.client.mqttv3-1.1.1.jar) @ central (http://repo1.maven.org/maven2/, default, releases+snapshots)
    [package cl-watson-iot]

    The entire project is available in this GitHub repository; the main function is mqtt-connect, included here to comment on a few details:

    (defun mqtt-connect (topic broker-url client-id &optional username password)
    "Establishes a MQTT connection to TOPIC; returns the mqtt client object"
    (let ((mqtt-conn-options (jss::new 'MqttConnectOptions))
    (mqtt-client (jss::new 'MqttClient broker-url client-id))
    (ssl-context (#"getInstance" 'javax.net.ssl.SSLContext "TLSv1.2")))
    (#"setCleanSession" mqtt-conn-options jss::+true+)
    (#"setKeepAliveInterval" mqtt-conn-options 30)
    ;; When username and password are provided use them
    (when (and username password)
    (#"setUserName" mqtt-conn-options username)
    (#"setPassword" mqtt-conn-options (#"toCharArray" password)))
    ;; For Watson IoT we need to add TLS1.2
    (#"init" ssl-context jss::+null+ jss::+null+ jss::+null+)
    (#"setSocketFactory" mqtt-conn-options (#"getSocketFactory" ssl-context))
    (#"setCallback" mqtt-client (mqtt-callback))
    (#"connect" mqtt-client mqtt-conn-options)
    (print (format nil "Connected to ~A" broker-url))
    mqtt-client))

    We are using the JSS notation feature of ABCL, one of the several different options in terms of Java interoperability; the code above is creating new Java objects, using instance methods, etc. The entire code contains an example of how to implement a Java interface to function as a callback.

    final

    The send-loop function sends 20 messages with an interval of 2 seconds; the values are random numbers between 0 and 50.

    random_cl

     

    And this is it: we have added Java dependencies to our Common Lisp project and used ABCL to send data to our Watson IoT instance.

  8. Conclusion

    This recipe shows how we can use ABCL in order to leverage existing Java libraries; many times that usage can be relatively well contained and the vast majority of the code can be ANSI Common Lisp, portable to all implementations, while at the same time taking advantage of the extensive library support of Java.

Join The Discussion