Kubernetes with OpenShift World Tour: Get hands-on experience and build applications fast! Find a workshop!

Integrating artificial intelligence into your IoT solutions

In this article, you learn how to use artificial intelligence, or at least machine learning, to raise the alarm when there are changes in a supposedly static environment, such as a hay barn while the hay is drying after the harvest. I use two methods to achieve this: visual recognition and image comparison.

Visual recognition requires more processing than can be done easily on a Raspberry Pi. The solution here is to upload pictures of the IBM Cloud, and ask IBM Watson Visual Recognition to identify the objects in them. If a new object appears, or if an expected object disappears (and doesn’t show for a whole day, because objects may only be identifiable under certain lightening conditions), this AI system raises an alarm.

Because object recognition using IBM Watson Visual Recognition requires significant bandwidth to upload pictures to the IBM Cloud, I designed an AI system that can work on a low-bandwidth network connection, such as a LoRa connection. To detect changes in such an environment, the second part of this article uses image comparison. Images are taken every ten minutes, and each time the image is compared to the image taken 24 hours prior. This way, the changes in lightening conditions will hopefully be minor enough to prevent false alarms.

What you’ll need to build your application

  • A Raspberry Pi
  • A Raspberry Pi Camera Module
  • An HDMI monitor.
  • USB keyboard and mouse (wireless or USB wired, it does not matter)
  • A micro SD card (8 GB is more than enough)

Architectures

Short-range architecture

To implement visual recognition in the cloud, we will base our architecture on the short range architecture in the first article. The devices in the hay barn use WiFi to communicate with an access point in the farmhouse which is connected to the Internet. In addition to the NodeMCU and DHT11 sensors from that article, the hay barn has a Raspberry Pi with a Raspberry Camera module to detect objects.

Figure 1: Short-range architecture

Short-range architecture, with visual recognition in the cloud

Note that the long range architecture is not appropriate for this use case because of data rate issues. The pictures taken by the Raspberry Camera are about 3.5 MB each. LoRa has a data rate of about 50 kbps. 3.5 MB is 3584 kB (a megabyte is 1024 kilobytes), or 28,672 kb (eight bits per byte). So it would take about 570 seconds, which is close to ten minutes, to transfer a single image. This is excessive, especially in an environment where we need to transfer other information, such as sensor readings.

So, in my solution, the Raspberry Pi uses the Internet to access two IBM Cloud services: Object Storage and Cloud Functions. After it takes a picture, the Raspberry Pi uploads it to IBM Cloud Object Storage. It then calls an action on IBM Cloud Functions to inform it about the new picture. The action then accesses IBM Watson Visual Recognition service to identify the objects and an IBM Cloudant Database to compare the current objects to the objects it has seen in the past. If it detects a significant change, it uses SendGrid to send a message to a human being.

We could have used the IBM Watson IoT Platform to store the pictures, but it would require us to handle additional complexity (register the Raspberry Pi as a device) and does not provide us with additional value. The sensors use the IoT Platform because it’s a highly available infrastructure to upload and store sensor readings. However, IBM Cloud Object Storage is also highly available infrastructure, and in contrast to the IoT Platform, it is optimized for larger information such as the pictures captured by the camera.

Long-range architecture (on the edge)

To implement visual recognition on the edge, we will base the architecture on the long range architecture in the first article. The one difference is the addition of a Raspberry Pi with a camera. This Raspberry Pi connects to the Barn ESP32 using WiFi, and the Barn ESP32 will send out alarms using LoRa to the Farmhouse ESP32 to the IBM IoT Platform.

Figure 2: Long-range architecture

Long-range architecture

The source code

I use an iterative development model. Several versions of the software are available on GitHub.

The first set of programs are used to build the visual recognition in the cloud:

  • 31_take_picture.js: Node.js code to run on the Raspberry Pi to take a picture with the Raspberry Camera.
  • 32_upload_picture.js: Adds the code to upload the picture to IBM Cloud Object Storage.
  • 33_visual_recognition.js: Adds the code to call IBM Watson Visual Recognition to identify the objects in the picture. For the sake of simplicity, at this point the entire program is still running on the Raspberry Pi.
  • 34_classifier_output.json: Not a program, but an example of the output produced by IBM Watson Visual Recognition.
  • 35_action.js: An IBM Cloud Functions action that calls IBM Watson Visual Recognition and uses IBM Cloudant Database to identify changes.
  • 36_raspberry_final.js: The final version of the Raspberry Pi Node.js program. Instead of calling IBM Watson Visual Recognition, it calls the IBM Cloud Functions action.
  • 37_action_with_email.js: The final version of the IBM Cloud Functions action, which also includes calling SendGrid to send an e-mail to a human if necessary.

The final two programs are used to build the visual recognition on the edge:

Implementing visual recognition in the cloud

This architecture uses IBM Watson Visual Recognition. To do this we take our pictures locally using a Raspberry Pi, but then use the Internet to upload the pictures to the IBM Cloud and then the AI in IBM Watson Visual Recognition to identify the objects in the picture. Once we identify the objects, we can compare them to previously seen objects from that same camera.

1

Set up the Raspberry Pi

For now, connect the Raspberry Pi to your own network to set it up. When you are ready to deploy, change the network parameters as needed and connect it in the hay barn.

1a Set up the Raspberry Pi, the Camera module, and the operating system

These are the steps to configure the Raspberry Pi to make it usable as a surveillance camera.

  1. Download and install NOOBS (the lite version is sufficient) on the microSD card.
  2. Insert the microSD card into the Raspberry Pi. Connect the HDMI, keyboard, mouse, and power (the micro-USB connection).
  3. Connect the Raspberry Pi Camera module.
  4. Configure the wifi network to connect to the Internet.
  5. Select Raspbian and click Install (you only need the graphics to verify the camera is taking pictures, if you are going to copy the image files and open them from elsewhere, you can select Raspian Lite instead).
  6. After the installation reboot the Raspberry Pi.
  7. When you see the login prompt, log in as pi with the password raspberry. Note that there is no need to change the wifi settings – the wifi values you gave NOOBS are preserved.
  8. Change the default password: passwd pi.
  9. Enable sshd, as explained here.
  10. Use an SSH client (my favorite on Windows is Bitvise) to connect to the Raspberry Pi. You may need to connect to your router to identify the IP address.

1b Install the picturing taking software on the Raspberry Pi

With the hardware and the operating system running, the next step is to download the software that we will need.

  1. Install NodeJS. This environment allows us to program in JavaScript. Note that while you can install NodeJS using an apt-get from the standard repository, that version is very old.

    sudo apt-get update
    sudo apt-get dist-upgrade
    curl -sL https://deb.nodesource.com/setup_10.x | sudo \
    -E bash –
    sudo apt-get install -y nodejs
    node -v
    
  2. Install npm, the NodeJS package manager.

     sudo apt-get install npm
    
  3. Install the NodeJS modules this project needs.

     npm install pi-camera ibm-cos-sdk watson-developer-cloud
    
  4. Download and run the program.

  5. In a browser on the Raspberry Pi, open file:///tmp/test.png to see the picture you just captured.

The first line creates an object for the pi-camera module. This reference is not supposed to change (it will always point to the same object), so we can declare it using the const keyword.

const PiCamera = require('pi-camera');

Note: The const keyword does not protect the value of the object in the variable. To verify this statement, run this line:

const c = {a:1}; c.b=2; console.log(c);

These lines specify the object for taking the picture:

const myCamera = new PiCamera({
    mode: 'photo',
    output: '/tmp/test.png',
    nopreview: true
});

Finally, this code snaps the picture:

myCamera.snap()
    .then(result => console.log('Success ' + result))
    .catch(err => console.log('Error ' + err));
2

Upload the images

Before we can do anything with the images, we need to make them accessible to services on the Internet. IBM Cloud Object Storage gives us a place to do that.

2a Create an IBM Cloud Object Storage bucket

  1. Log on to the IBM Cloud console.
  2. Click Create resource.
  3. Select Storage > Object Storage.
  4. Scroll down to name the service BarnImages and click Create.
  5. Click Create Bucket.
  6. Name the bucket and click Create bucket (I selected images4detection, but you might need a different name).
  7. Create a service credential for the Raspberry Pi.
    1. Click Service credentials in the left sidebar.
    2. Click New credential.
    3. Name the credential RaspberryUploadCred.
    4. Select Select Service ID > Create New Service ID.
    5. Name the service ID RaspberryUploadCred.
    6. Click Add.
    7. Click View credentials and copy the credential to the clipboard.
  8. Click Buckets on the left sidebar and then images4detection.
  9. Click Endpoint.
  10. Select the appropriate location (us-geo, eu-geo, or ap-geo)
  11. Copy one of the public end points. You will need it in your code.

2b Upload the picture from the Raspberry Pi

Download this program, replace the serviceAcctCred, bucket, and storageEndpoint constants with your own values.. Run the program, then go to the storage you created. There should be a picture.png object there, download and view it – it is the picture the Raspberry Pi took.

This is the code to connect to the IBM Cloud Object Storage using this library.

const ObjectStorageLib = require("ibm-cos-sdk");
const objectStorage = new ObjectStorageLib.S3({
    endpoint: storageEndpoint,
    apiKeyId: serviceAcctCred.apikey,
    ibmAuthEndpoint: 'https://iam.ng.bluemix.net/oidc/token',
        serviceInstanceId: serviceAcctCred.resource_instance_id
});

The camera control code writes the picture to a file (it is a wrapper around the command line interface). So we need the filesystem module to read the picture from the file.

const fs = require("fs");

There are two ways to put an object on IBM Cloud Object Storage. The simpler is to do a single part upload, but that is limited to 5 MB. There is a more complicated method that uploaded the file in multiple chunks, but luckily we don’t need it here because the images are less than 5 MB.

// The image file size is less than 5 MB, so there's no need
// for a multi-part upload

The file name, bucket name, and object storage configuration are constants. However, the key could vary. If we have multiple Raspberry Pi instances running, we may want to distinguish between different sources of pictures. If we preserve the pictures, we might also want to distinguish them by time.

const uploadImage = (key, callback) => {
    fs.readFile(fname, (err, data) => {
        if (err) {

Read the file. If you fail, abort the process. This condition shouldn’t happen.

             console.log(`Error reading file ${fname}: ${err}`);
             process.exit();
        }

Put the object in the IBM Cloud Object Storage. The data is the data read from the file.

        objectStorage.putObject({
             Bucket: bucket,
             Key: key,

The access control list allows anybody to read the information. This way, we won’t need to share any authentication token with IBM Watson Visual Recognition, which simplifies the program.

             ACL: "public-read",
             Body: data
        }).promise()
        .then(callback)

It is more important to handle errors here gracefully. This kind of error is a lot more likely because if there’s a problem with the Internet connection, for example, we will be unable to upload.

        .catch(e => console.log("Image upload error: " + e))
    });
};

Snap the picture and then upload it (if successful).

myCamera.snap()
    .then(uploadImage("picture.png", () => {
        console.log("Done");

For some reason the IBM Cloud Object Storage library keeps the application running after it finishes the upload. To avoid this, call process.exit() to exit.

        process.exit();
    }))
    .catch(err => console.log('myCamera.snap error ' + err));

2c Download the picture from the URL

According to the documentation, the URL to read an object is https://<endpoint>/<bucket-name>/<object-name>. Browse to the appropriate URL for your picture. Note that the browser doesn’t know it is a picture, because the Content-Type is not set – so it is considered a file and just saved.

Classify objects in the image

Now that we have the image from the picture on the cloud, the next step is to actually use IBM Watson Visual Recognition.

  1. Log on to the IBM Cloud console.
  2. Click Create resource.
  3. Select AI > Visual Recognition.
  4. Name it barnImages and click Create.
  5. Click Show to see the API key, and copy the credential.
  6. Copy this program, and edit it to put in your own credentials (both serviceAcctCred and visualRecognitionCred) as well as the storage endpoint (storageEndpoint).
  7. Run the program, and see the result.

Image classification is also available through an API. Here is how we use it:

The first step is to get the object class for the API. The same library also includes other APIs that are part of IBM Watson.

const VisualRecognitionV3 =
require('watson-developer-cloud/visual-recognition/v3');

Next, create the actual API object. To account for future upgrades, the API object needs to know what version of the API to use (at writing, the latest is from March 19th, 2018). It also needs the API key to know which account to use, and bill if necessary.

const visualRecognition = new VisualRecognitionV3({
    version: '2018-03-19',
    iam_apikey: visualRecognitionCred.apikey   
  });

This is the code that takes the picture, uploads it, and then calls the visual recognition API.

myCamera.snap()
    .then(uploadImage(objName, () => {
        console.log("Done uploading");

The parameters for visual recognition. We only use one, the URL where the image is available. There are a few others you might find useful.

        const params = {
            url:
    `https://${storageEndpoint}/${bucket}/${objName}`
        };

        console.log(`Picture located at ${params.url}`);

This code actually calls the classifier. The response is provided in JSON format.

        visualRecognition.classify(params, (err, response) => {
            if (err)
                console.log(err);
            else
                console.log(JSON.stringify(response,
                  null, 2));

            process.exit();
        });
    }))
    .catch(err => console.log('myCamera.snap error ' + err));

You can see sample JSON produced by image classification here. In the case of a single image, the classes of the objects discovered are available at .images[0].classifiers[0].classes.

4

Detecting changes

The purpose of the camera and the visual classification is to identify when something changes in the hay barn, which requires informing a human being. Here is an algorithm that identifies such changes, while not being affected by the fact most hay barns get light only by sunlight, which changes angle during the day and does not shine at all at night.

  1. Periodically, the Raspberry Pi takes a picture and uploads it to IBM Cloud Object Storage.
  2. The Raspberry Pi informs the server code, running on IBM Cloud Functions.
  3. The server code calls IBM Watson Visual Recognition to identify the objects.
  4. The server code uses the IBM Cloudant Database to compare the objects identified to those seen in the recent past.
  5. If objects are added or deleted, inform a human being, and include the uploaded picture
  6. If no objects changed, erase the picture to save storage space
  7. Update the IBM Cloudant Database as needed.

4a Configure the IBM Cloudant database

The next step is to configure the data storage and the server code that uses it.

We need a database to store the objects identified on each device.

  1. Log on to the IBM Cloud console.
  2. Click Create resource.
  3. Select Databases > Cloudant.
  4. Name the service barnImages and select the authentication method Use both legacy credentials and IAM.
  5. Click Create.
  6. After the Cloudant database is provisioned, click Services > barnImages in the resource list.
  7. Create a service credential the database.
    1. Click Service credentials in the left sidebar.
    2. Click New credential.
    3. Name the credential deviceObjects.
    4. Leave the role as Manager.
    5. Click Add.
  8. Click View credentials and copy the credential to the clipboard.
  9. Click Manage in the left sidebar and then LAUNCH CLOUDANT DASHBOARD.
  10. Click Databases in the left sidebar (or the icon: ).
  11. Click Create Database, name the database device_objects, select Non-partitioned and click Create.

4b Configure IBM Cloud Functions

The server code runs in IBM Cloud Functions. This way, we are only using resources when we actually need them. We are only going to create a single action to run the server code.

  1. Log on to the IBM Cloud console.
  2. Click Create resource.
  3. Select Compute > Serverless Compute > Functions.
  4. Click Start Creating.
  5. Click Actions and then Create.
  6. Click Create Action.
  7. Create a package for the action:
    1. Click Create Package.
    2. Name the package barnImages and click Create.
  8. Name the action processBarnImage.
  9. Verify that the runtime environment is Node.js 10 or a later version.
  10. Click Create.
  11. Replace the default program with the program from GitHub. The program is explained below, together with the Raspberry Pi code that calls it.
  12. Click APIs in the left sidebar.
  13. Click Create Managed API.
  14. Name the API Process Barn Images.
  15. Click Create operation.
  16. Create an operation with these parameters:

    • Path: /image
    • Verb: GET
    • Package containing action: barnImages
    • Action: processBarnImages
    • Response content type: application/json
  17. Leave the options on their defauilt values and click Create.

  18. Copy the route to be able to use it later.

Parts of the IBM Cloud Function code are identical to what we have done with Node.js on the Raspberry Pi. Here I explain the parts that are new.

We connect to the IBM Cloudant Database. When supporting legacy credentials, the URL includes the user name and password. This is simpler than using the IAM credential.

// Credential for the IBM Cloudant database
const cloudantCred = {
… redacted …
}

const cloudant = require('@cloudant/cloudant')({
    url: cloudantCred.url
});

Next we specify the database to use.

const database = "device_objects";

const db = cloudant.db.use(database);

The main function receives in params a single parameter, the name of the object uploaded to the IBM Cloud Object Storage. This name includes both the deviceID and timestamp.

const main = params => {

This code parses the object name. It is pict_<device id>_<timestamp>.png, so first we remove the extension and then split it on the underline (_) character.

// Parse the object name to get the information encoded in it
const splitObjName = params.objName.split(".")[0].split("_");
const devID = splitObjName[1];
const timestamp = splitObjName[2];

When an IBM Cloud Function cannot return a result immediately it returns a Promise object, which includes a function for the system to call. This function itself takes two functions, success and failure, to be called if the process succeeds or fails respectively.

In our case, we can expect at least three slow processes: recognize objects in the image, reading the current database entry for the device, and writing the updated entry. This way, we can run them without tying up too many resources.

// Classification takes time, so we need to return a
// Promise object.        
 return new Promise((success, failure) => {

This is how we recognize objects, same as we did on the Raspberry Pi.

visualRecognition.classify(visRecParams,
    (err, response) => {
  if (err) {

In the case of error, call the failure function. This function expects a structure, here we provide the error location and the exact error. It is useful to provide the error location because there are multiple steps that can go wrong resulting in failure being called.

failure({
  errLoc: "visualRecognition.classify",
  err: err
});
  return;  // Exit the function
}

We only use the object names, not the degree of confidence or the location in the hierarchy of object types, so remove that information leaving an array of object names.

const objects = response.images[0].classifiers[0].classes.map(x => x.class);

Get the entry for the device. It is an associate array of object names and the last time they were recogmized in the image from that device.

db.get(devID, (err, result) => {

Error status 404 means there is no entry for the device. This is the expected result when the device is new, so handle it – create the object and insert it into the database.

// Not really an error, this is just a new device without an entry yet
                if (err && err.statusCode == 404) {
                    var dataStruct = {};  // Data structure
// to write to database
                    objects.map(obj =>
dataStruct[obj] = timestamp);

                    db.insert({_id: devID, data: dataStruct},
 (err, result) => {
                        if (err) {
                            failure({
                                errLoc: "db.insert (new entry)",
                                err: err
                            });
                            return;  // Exit the function
                        }

If we want the user to be informed that a new device has been added, we can do it here.

// Send message to the user here

Call the success function. We ran successfully, there is just no information to report because there is no previous state to have changes from.

    success({});

}); // end of db.insert

At this point, we are inside the db.get callback. We want to exit the function to stop further processing, which is not needed if this is a new device.

    return;
}

If db.get failed with any other error, it is a real error and we need to report it as such.

// If we got here, it's a real error
if (err) {
  failure({
    errLoc: "db.get",
    err: err
  });
  return;  // Exit the function
}

For the sake of readability, the rest of the processing is in a different function, compareData.

        // Compare the old data with the new one,
        // update as needed and update the database
        // (also, inform the user if needed)
        compareData(objects, result,
          timestamp, success, failure);
    });   // end of db.get
  });  // end of visualRecognition.classify
 });  // end of new Promise
}; // end of main



// Compare the old data with the new one, update as needed
// and update the database
// (also, inform the user if needed)
const compareData = (objects, result, timestamp,
        success, failure) => {

This is not the functional programming ideal, but we build the update data to write to the database and the lists of new and missing objects arrays in these variables.

var data = result.data;  
var newObjects = [];
var missingObjects = [];
var informHuman = false;

Iterate over the list of objects from IBM Visual Recognition. If an object is new, add it to the list of new objects. Update the time it was last seen to the timestamp.

objects.map(object => {
        // The object is new, insert it to the new object list
        if (!result.data[object])
            newObjects.push(object);

        // Either way, update the time it was last seen
        data[object] = timestamp;
    });

Now, iterate over the list of objects that the database has, and see if any have last been seen over 24 hours ago. If so, add them to the missing objects list and delete their timestamp (so they won’t appear missing in every future invocation until they appear again).

// Look for objects that haven't been seen in 24 hours
const deadline = new Date(new Date(timestamp) – 24*60*60*1000);
Object.keys(data).map(object => {
  const lastSeen = new Date(data[object]);
  if (lastSeen < deadline) {
    missingObjects.push(object);
    delete data[object];
  }
});  // end of Object.keys(data).map

If there are new objects or missing ones, we need to inform a human being. We implement this feature later in the article.

// Do we need to inform a human?
if (newObjects.length > 0 || missingObjects.length > 0)
  informHuman = true;

Note that to update a Cloudant database document we need the entry the document revision (_rev). This requirement allows Cloudant to maintain consistency by rejecting changes based on stale data.

// Update the data in the database (that will always
// be required
    // because if nothing else the timestamps changed)
    const newEntry = {
        _id: result._id,
        _rev: result._rev,
        data: data
    };

    db.insert(newEntry, (err, result) => {

        if (err) {
            failure({
                errLoc: "db.insert",
                err: err
                });
            return;  // Exit the function
        };

        success({
            new: newObjects,
            missing: missingObjects,
            data: data
        });
    });  // end of db.insert
}; // end of compareData

4c Classifying objects

Copy the final version of the Raspberry Pi from GitHub. Set APIUrl to the route for the API you created on IBM Cloud Functions in the previous step. Set serviceAcctCred to the IBM Cloud Object Storage credential. Modify storageEndpoint to the correct value for your location. If you want an update frequency different from every ten minutes, change the freq value.

This is the final version of the Raspberry Pi program. It takes pictures, uploads them, and then calls on server code for further processing. Most of this program is identical to what we have already done earlier on the Raspberry Pi. Here I explain the parts that are new:

The code that processes the picture is now wrapped in a function to make it easy to call.

const processPicture = () => {
    ...

            uploadImage(currentObjName, () => {
                console.log("uploaded " + currentObjName);

After the upload finishes, use https.get to call the IBM Cloud Functions code. We could have used POST and written the parameters in the HTTP header, but when there are only a few parameters it is easier to send them as a query string. Here there is only one, the name of the uploaded object.

https.get(`${APIUrl}${funcPath}?objName=${currentObjName}`,

This callback, called after we receive a response from IBM Cloud Functions, is not really necessary. I only put it here for debugging purposes, so know that the server code ran for this object.

        (res) => {
        console.log(
    `Done with ${currentObjName}`);
        console.log("-----------------------");
    }); // end of https.get
...
};  // end of processPicture

Run processPicture once immediately, and then run it again every few minutes (determined by the freq variable). JavaScript uses milliseconds to track time, so multiple the number of minutes by 60,000.

processPicture();

// Process a picture every freq minutes
setInterval(processPicture, 1000*60*freq);
5

Inform a human

The one feature we are still missing is how to inform a human being when an object is either new or missing for the last 24 hours. To do that, we can use the e-mail service from SendGrid:

  1. Browse to https://sendgrid.com/free/.
  2. Click Try for Free and create an account for yourself.
  3. Find the Integrate using out Web API tile and click Start.
  4. Choose the Web API.
  5. Choose cURL. Our actions use the Node.js runtime environment, but it is not trivial to add another library to IBM Cloud Functions. We can add libraries, but it is somewhat complicated. If you want to learn how to do this, read about the development toolchain in this article.
  6. Name your API key (for example, informPeople) and click Create Key.
  7. Copy the key from the text field below the Create Key button.
  8. Go to your e-mail and confirm your account.

This action calles the SendGrid service directly using HTTPS.

This function converts a list into an HTML unordered list. It uses the map function to convert a list items ([“it“, “ems“], for example) into a list of HTML list ([“

  • it
  • “, “
  • ems
  • “]). Then, it uses the reduce function to turn that list into a single HTML expression (“<ul><li>it</li><li>ems</li></ul>“).

    const list2HTML = (lst) => {
        if (lst.length == 0)
            return "None";
    
        return `<ul>${lst.map(x => "<li>"+x+"</li>").
    reduce((a,b) => a+b, " ")}</ul>`;
    };
    

    This function creates the entire HTML to send in the body of the message. This includes the changes themselves, as well as the picture that shows them.

    const makeMsg = (newObjs, missingsObjs, pictureURL) => {
        return `
            <H2>Changes</H2>
            <h4>New objects:</h4>
            ${list2HTML(newObjs)}
    
            <h4>Missing objects:</h4>
            ${list2HTML(missingsObjs)}
    
            <H2>Picture</H2>
            <img src="${pictureURL}" height="500" width="800">
        `;
    };   // makeMsg
    

    This function sends an e-mail to the user with all the information about a change in objects that the server identified:

    const informUserFunc = (newObjs, missingObjs,
      devID, pictureURL, success) => {
    

    To use the SendGrid API call to send an e-mail, we send an HTTPS POST request. The headers need to include the content type and our authotrization to use the service.

    const req = https.request({
            method: "POST",
            hostname: "api.sendgrid.com",
            path: "/v3/mail/send",
            headers: {
                "Content-Type": "application/json",
                "Authorization": `Bearer ${SendGridAPIKey}`
            }
        },   // end of options for https.request
    

    This is the function called after the request is processed. It is the final step of the action for this picture.

    (res) => {
            success({loc: "informUserFunc",
                    statusCode: res.statusCode,
                    statusMessage: res.statusMessage
            });
        });  // end of https.request
    

    This is the object that SendGrid expects. The content.value field is the HTML, created by makeMsg.

    const mailToSend =
            {"personalizations": [{"to": [{"email": destEmail}]}],
                "from": {"email": sourceEmail},
                "subject": `Changes in device ${devID}`,
                "content": [{
                    "type": "text/html",
                    "value": makeMsg(newObjs, missingObjs, pictureURL)
                }]
            };
    

    This is the code to write the JSON for the mail object to the body of the POST request. Because it is the body and not a header field, it is here and not in the https.request call above.

        req.write(JSON.stringify(mailToSend));
        req.end();
    
    }; // end of informUserFunc
    
    6

    Prepare for deployment

    This is a prototype written to teach, not a fully featured program written to actually deploy. Here are a number of features you should add if you plan to actually deploy this in production:

    1. Delete unused pictures. If the visual recognition does not identify any need to inform the user, delete the picture object from IBM Cloud Object Storage using deleteBucket. Storage is cheap, but not free.
    2. Identify when a device is offline. If there is no message from a device for a certain amount of time, send an alert. To do this, you can use a periodic trigger to run an action that reads all the IBM Cloudant Database entries and checks the date of latest update for each. If an entry has not been updated in over an hour, for example, inform a user.
    3. Give devices a usable identifier. Use a separate database on Cloudant to convert between MAC address identifiers, such as b8:27:eb:5f:aa:73, and useful identifiers, such as “Barn #5, the one next to the Creek”.

    Implement visual recognition on the edge

    In this series, we also use a long distance architecture, which uses LoRa for network connectivity. In that case it is impossible to upload pictures. It is possible to run the Tensor Flow library to identify objects on a Raspberry Pi. However, that is a very complicated process. Even after you complete it, the low RAM of the Raspberry Pi (maximum 1 GB) limits the visual recognition models you can use and therefore the accuracy of detection.

    A more appropriate solution is to take pictures and compare them to previous pictures using structural similarity. Doing this compensates for different lightening conditions (sunny vs. cloudy vs. rainy) while still identifying significant changes.

    1

    Setting up the Raspberry Pi

    The setup is nearly identical to Raspberry setup in the For now, connect the Raspberry Pi to your own network to set it up. When you are ready to deploy, change the network parameters as needed and connect it in the hay barn. If you use the configuration from the first artile, the SSID is barn-net and there is no password.

    1a Set up the Raspberry Pi, the Camera module, and the operating system

    These are the steps to configure the Raspberry Pi to make it usable as a surveillance camera.

    1. Download and install NOOBS (the lite version is sufficient) on the microSD card.
    2. Insert the microSD card into the Raspberry Pi. Connect the HDMI, keyboard, mouse, and power (the micro-USB connection).
    3. Connect the Raspberry Pi Camera module.
    4. Configure the wifi network to connect to the Internet.
    5. Select Raspbian and click Install (you only need the graphics to verify the camera is taking pictures, if you are going to copy the image files and open them from elsewhere, you can select Rasberry Lite instead).
    6. After the installation reboot the Raspberry Pi.
    7. When you see the login prompt, log in as pi with the password raspberry. Note that there is no need to change the wifi settings – the wifi values you gave NOOBS are preserved.
    8. Change the default password: passwd pi.
    9. Enable sshd, as explained here.
    10. Use an SSH client (my favorite on Windows is Bitvise) to connect to the Raspberry Pi. You may need to connect to your router to identify the IP address.

    1b Install the picturing taking software on the Raspberry Pi

    With the hardware and the operating system running, the next step is to download the software that we will need.

    1. Install NodeJS. This environment allows us to program in JavaScript. Note that while you can install NodeJS using an apt-get from the standard repository, that version is very old.

       sudo apt-get update
       sudo apt-get dist-upgrade
       curl -sL [https://deb.nodesource.com/setup_10.x](https://deb.nodesource.com/setup_10.x) | sudo -E bash –
       sudo apt-get install -y nodejs
       node -v
      
    2. Install npm, the NodeJS package manager.

       sudo apt-get install npm
      
    3. Install the NodeJS modules this project needs.

       npm install pi-camera
      
    4. Download and run the program from https://github.com/qbzzt/IoT/blob/master/201801/Hay_Bale_Fire/31_take_picture.js. This program is explained above as part of the visual recognition architecture.

    5. In a browser on the Raspberry Pi, open file:///tmp/test.png to see the picture you just captured.

    2

    Comparing images

    The package we use for comparing images is img-ssim. . This is how we use it.

    1. Install the image comparison Node.js package. Note that this is a lengthy process. On a Raspberry Pi 3 Model B, it takes about seventeen minutes. This process compiles the C code to perform the similarity calculations. This produces a lot of warning messages, but you can ignore them.

       npm install img-ssin
      
    2. Download this file as compare_img.js and run it:

       node compare_img.js
       `
      

    Image comparison is a very complicated algorithm, but calling it is extremely simple. All you provide is a couple of file names to compare and a calllback function with two parameters: an error and a similarity score. The similarity score is between zero and one.

    imgSSIM(
        "/tmp/pict1.png",
        "/tmp/pict2.png",
            (err, similarity) => console.log(err || similarity);
    );  // end of imgSSIM
    
    3

    Sending alarms after comparing images

    The pictures are going to look different at different times of the day. A hay barn is unlikely to have artificial lighting, and sunlight naturally comes in different angles. This could be a problem, but the solution is to take pictures at the same time of day.

    This program is designed to run every ten minutes. Each time it takes a picture, and if there’s a picture from the previous day it compares the two and raises the alarm if they are sufficiently different. Either way, it replaces the old picture with the new one. Doing so allows us to compensate for seasonal changes in sunlight intensity and direction.

    The program starts with configuration parameters. The first one, timeDiff, specifies that the 24 hour period is divided into 144 ten minute slices. Two pictures are compared if they are from the same time slice.

    // Divide time into ten minute slices
    const timeDiff = 10*60*1000;
    

    This is the similarity score below which the program decides the change is sufficient to raise an alarm.

    // Threshold at which there's an alarm
    const threshold = 0.75;
    

    The directory to store the pictures.

    // Directory to store pictures
    const pictDir = "/tmp";
    

    And the device identifier for this particular barn. It is a negative number to distinguish it from the NodeMCU sensors that are always positive.

    // Identifier for this device
    const devID = -1;
    

    The next part of the program is the libraries required. In addition to node-webcam and img-ssim, we need the fs module. This module allows us to manipulate files, such as pictures. We use it to replace old pictures with new ones. To send alerts to the ESP32 and from there through LoRa to the Internet we need the HTTP library for Node.js.

    const NodeWebcam = require( "node-webcam" );
    
    // Takes about 20 minutes to install
    const imgSSIM = require("img-ssim");
    
    // File manipulation
    const fs = require("fs");
    
    // HTTP client code
    const http = require('http');
    

    This function gets the time stamp for the current slice.

    // Get a timestamp for the current slice of time
    const getTimeStamp = () => {
        const now = Date.now();   // msec from start of epoch
        const dateObj = new Date();
    

    Using Math.floor(x/y)*y we get the value of the highest multiple of y that is still smaller than x. In this case, it’s the time (in milliseconds since the beginning of the epoch) that the current time slice started.

    dateObj.setTime(Math.floor(now / timeDiff) * timeDiff);
    

    The time stamp string is <hour>_<minute>. I tried to use the traditional separator, a colon (:), but it confuses the image comparison library. It can accept pictures in URL format (for example, https://upload.wikimedia.org/wikipedia/commons/thumb/5/51/IBM_logo.svg/800px-IBM_logo.svg.png). So it interprets /tmp/pict_1:20.png as a URL with the scheme /tmp/pict_1. It does not know what to do with that scheme.

        return `${dateObj.getHours()}_${dateObj.getMinutes()}`;
    };
    

    The next two definitions are for file names. The getFname function provides the file name for the current time slice, and newPictFname is the name for the picture we take. It cannot be the time slice name, because then it would delete the existing picture and we will be unable to compare them.

    // Get the filename for a picture taken at this time
    const getFname = () => `${pictDir}/pict_${getTimeStamp()}.png`;
    
    // The filename of the new picture, the one in processing
    const newPictFname = `${pictDir}/new_pict.png`;
    

    This function, processPicture, does most of the work of the program. It takes the new picture, compares it with the old one, and renames the new picture to overwrite the old one. If the two are sufficiently different, it also writes a message to the console.

    const processPicture = () => {
    

    The return value of getFname() could change which the function is running, because it is time dependent. This way we have a consistent value.

    const currentPict = getFname();
    
    // Take new picture
    Webcam.capture(newPictFname, (err, data) => {
      if (err) {
        console.log(`Webcam.capture error: ${err}`);
        return ;  // Exit the function call
      }
    

    The fs.existsSync function checks if a file exists. The postfix -Sync means it is a synchronous function, one that returns the result instead of running a callback. In a server process using such a function is typically a bad idea because it blocks all other requests. However, in this case the program is only doing one thing, and it cannot continue doing it until it has the result – so there is no need for the additional complexity of a callback.

    if (fs.existsSync(currentPict)) {
    

    If the file exists, compare the new image with the old one.

    imgSSIM(currentPict, newPictFname, {
      enforceSameSize: false,
      resize: true
    }, (err, score) => {
      if (err) {
        console.log(`imgSSIM error: ${err}`);
        return ;
      };
    

    If the two images are not similar, raise the alarm. For the alarm to be meaningful, it needs to be communicated through LoRa to the Internet.

    The way the long range architecture uses LoRa is to have an ESP32 access point in the barn, which receives HTTP requests on http://10.0.0.1///, with all three values being decimal numbers (see https://www.ibm.com/developerworks/library/iot-lpwan-lora-nodemcu-dhtsensors/index.html, step 2c). These values are then communicated through LoRa to a farmhouse ESP32, and from there to an IBM Watson IoT Platform (which may be connected to a Node-RED application, see Part 3 of this series).

    It is much simpler not to have to modify the ESP32 programs. The NodeMCU chip ID is always a positive number. We can use negative chip ID numbers to identify the different barn Raspberry Pi’s. Then, on the cloud, we can distinguish the negative chip IDs and treat them differently, by raising an alert any time there is a sensor reading from them.

    // If the score is too low, raise the alarm.
    if (score < threshold)
    

    The http.get function actually informs the ESP32. The temperature and humidity are both 99 to ensure that even if the application on the cloud hasn’t been updated an alarm will be raised. Such a temperature (almost boilng) and humidity (almost full), is an alarm situation. We have no need for the result, so we don’t use a callback function.

    http.get(`http://10.0.0.1/${devID
      }/99/99`);
    

    We need to replace the old picture in currentPict with the new one. This operation is called twice: here and in the else section for fs.existsSync(). It is tempting to just put it outside of the if statement (whether the file currently exists or not we need to rename), but the two rename operations are called at very diffrerent time. The one here is in the callback function of the image comparison, and is only called after the image comparison is done with the old picture. The other one is when there is no old picture to compare, so it is right after the new picture is taken.

              // Replace the picture with the new one.
              fs.rename(newPictFname, currentPict, (err) =>
              {
                if (err)
                  console.log(`Rename to   ` +
                    `${currentPict} error: ` +
                   ` ${err}`);
              });  // fs.rename
    
            });  // imgSSIM
          } else {  // if (fs.existsSync())
                fs.rename(newPictFname, currentPict, (err) => {
                  if (err)
                    console.log(`NO ${currentPict} yet,` +
                      ` rename error: ${err}`);
                });  // fs.rename
          } // if (fs.existsSync())
    
        });  // Webcam.capture
    };  // processPicture definition
    

    Finally, run the processPicture function.

    processPicture();
    

    To run this program every ten minutes, run crontab -e and add this line:

    */10 * * * * /usr/bin/node.js ~pi/<directory>/app.js
    

    Conclusion

    In this article, you learned how to use a Raspberry Pi to take pictures. You learned two methods to identify changes in those picture.

    * The first method is to upload the pictures to IBM Cloud Object Storage, identify the objects using IBM Visual Recognition, use IBM Cloud Functions and IBM Cloudant Database to identify changes, and finally inform the user of such changes using SendGrid email.
    
    * The second method is to store pictures locally on the Raspberry Pi, and compare images from the same time on different days. The second method is less accurate information, but it can use low bandwidth connections that cannot upload pictures.
    
    Ori Pomerantz