Overview

Skill Level: Advanced

Learn about the benefits working with logical interfaces bring to application development.

Ingredients

You will need a Watson IoT Platform organization and an API key for that organization.

You will need to install the IBM Watson IoT Platform Python SDK.  This tutorial was created using version 0.9.4, you may find small differences in syntax and method signatures in latter versions as the SDK is still in active development.

In this tutorial we are going to use two of the reference device samples:

 

The problem we are faced with as application developers, and that logical interfaces are designed to resolve, is that if you want to write an application handling generic CPU and memory utilization data from connected devices, you would normally need to write code that understands the specific formats of both these devices.

The psutil device publishes events under the eventId psutil in the following format:

{
"name": "mysystem",
"cpu": 34.5,
"mem": 45.0,
"network": {
"up": 56.0,
"down": 12.7
}
}

The oshi device publishes similar data, under an eventId oshi in the following format:

{
"cpu": 72.55,
"memory": 45.0
}

Creating a logical interface involves three basic stages:

  • Define the physical device model for your device type(s).  This is the physical interface that your devices communicate their state to Watson IoT Platform.
  • Define one or more logical device model.  This is the logical interface that you applications consume device state from Watson IoT Platform.
  • Set up the mapping rules to translate from individual physical models of different device types to the shared logical model that all applications will use. 

 

Although it is possible to complete this tutorial without connecting any devices, it is strongly advised that before you proceed you connect at least one device of each type.  The clients can be ran on your local machine, or easily deployed on Docker.  If you do not connect these devices you will be unable to validate that the actions you have taken throughout this tutorial were successful.

The code snippets in this tutorial are all taken from the sample application code available on GitHub.  All the the scripts referenced in this sample require the user to set two environment variables: WIOTP_API_KEY & WIOTP_API_TOKEN

 

Step-by-step

  1. Create the event type schemas

    An event type schema is the definition of the message format for an event of a specific eventId.  The following code snippest shows how to create an event type schema from a local source file:

    import wiotp.sdk.appplication
    import json

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    schemaName="psutil-eschema"
    schemaFileName="psutil-schema.json"
    schemaDescription="Schema for PSUTIL event data"

    with open(os.path.join("schemas", "events", schemaFileName), "r") as schemaFile:
    schemaContent = json.load(schemaFile)

    schemaContentAsString = json.dumps(schemaContent)
    createdSchema = client.state.draft.schemas.create(schemaName, schemaFileName, schemaContentAsString, schemaDescription)
    print(createdSchema)

    Refer to configurePhysicalDeviceModel.py, you will notice we have packaged up the process of creating the schema into a manageSchema() method that adds some wrapper logic to make the operation idempotent, and we invoke it twice to create two schemas: psutil-eschema & oshi-eschema

    psutilSchema = manageSchema(
    schemaName="psutil-eschema",
    schemaFileName="psutil-schema.json",
    schemaDescription="Schema for PSUTIL event data"
    )
    oshiSchema = manageSchema(
    schemaName="oshi-eschema",
    schemaFileName="oshi-schema.json",
    schemaDescription="Schema for OSHI event data"
    )

    The schema definition files are available from the same repository:

     

    Action

    Using the code snippets above register the two schemas (psutil-eschema and oshi-eschema) in your organization before proceeding to the next step.

    More Information

    To create an event schema resource, we used the /draft/schemas API.  The schema definition file is passed to the Watson IoT Platform within a multipart POST (multipart/form-data). The body of the POST must contain at least two parts:

    • One with a name of schemaFile that contains the actual content of the file as the body of the part.
    • One with a name of name that contains a string that defines the name of the schema resource in the body of the part.

     

    Further reading: Schemas API documentation

     

  2. Create the event types

    In the previous step we registered & uploaded two schemas to Watson IoT Platform, they not not yet doing anything helpful though, because they first need to be associated with an Event Type. To create the Event Types we need the ID of the schemas we created in the previous step

    import wiotp.sdk.appplication
    import json

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    typeName="psutil-etype"
    description="PSUTIL Event Type"
    schemaId=createdSchema.id

    createdType = client.state.draft.eventTypes.create({
    "name": typeName,
    "description": description,
    "schemaId": schemaId
    })
    print(createdType)

    Refer to configurePhysicalDeviceModel.py, you will notice we have packaged up the process of creating the schema into a manageEventType() method that adds some wrapper logic to make the operation idempotent, and we invoke it twice to create two Event Types: psutil-etypeoshi-etype

    psutilEventType = manageEventType(
    typeName="psutil-etype",
    description="PSUTIL Event Type",
    schemaId=psutilSchema.id
    )
    oshiEventType = manageEventType(
    typeName="oshi-etype",
    description="OSHI Event Type",
    schemaId=oshiSchema.id
    )

    Action

    Using the code snippets above register the two event types (psutil-etype and oshi-etype) in your organization before proceeding to the next step.

    More Information

    To create an event type, we used the /draft/event/types API: Event Types API documentation

  3. Set up the physical interface

    In the previous step we used the schemas we uploaded to define Event Types, we now need to associate the event types with a physical interface.  Physical interfaces are re-usable, but in this example our two device types need different physical interfaces because the events they produce are not compatible.

    import wiotp.sdk.appplication

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    piName="psutil-pi"
    description="PSUTIL Physical Interface"

    createdPI = client.state.draft.physicalInterfaces.create({
    "name": piName,
    "description": description
    })

    eventId="psutil"
    eventTypeId=createdType.id

    createdPI.events.create({
    "eventId": eventId,
    "eventTypeId": eventTypeId
    })

    Again, refer to configurePhysicalDeviceModel.py, we have one again packaged up the process into a method, managePI() – that adds some wrapper logic to make the operation idempotent, and we invoke it twice to create two physical interfaces: psutil-pi & oshi-pi.

    Action

    Using the code snippets above register two physical interfaces (psutil-pi and oshi-pi) in your organization before proceeding to the next step.

    More Information

    To add an event type to the physical interface, we used the /draft/physicalinterfaces/{physicalInterfaceId}/events API: Physical Interfaces API documentation

     

  4. Update the device type

    The final step to complete the definition of our physical device model is to add the physical interfaces we have created to one or more device types.  Each device type can only have one physical interface, but a single physical interface can be implemented by multiple device types.

    import wiotp.sdk.appplication
    import json

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    typeId = "iotpsutil"

    deviceType = client.state.draft.deviceTypes[typeId]deviceType.physicalInterface = createdPI

    Action

    Using the code snippets above update the iotpsutil and iotoshi device types in your organization before proceeding to the next step.

    More Information

    To update a device type, we used the /draft/device/types/{typeId}/physicalinterface API: Device Types API documentation.

     

  5. Create the logical interface schemas

    With the physical device model complete, it is time to turn our attention to the logical device model.  We have now informed IBM Watson IoT Platform how devices will structure the data in the events they submit, by setting up a logical interface we will declare a model for how the service should make that information available to application.

    We are going to define two logical interfaces, iotpsutil devices implement both interfaces, iotoshi devices only implement the System Utilization interface.

    System Utilization Interfacesysutil-schema.json

    • cpu
    • memory

     

    Network I/O Interfacenetworkio-schema.json

    • network.outbound
    • network.inbound

     

    The code snippet below demonstrates how to create the logical interface schemas, it’s near-identical to the code that created the event type schemas previously:

    import wiotp.sdk.appplication
    import json

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    schemaName="sysutil-lischema"
    schemaFileName="sysutil-schema.json"
    description="Interface for anything capable of reading System Utilization data"

    with open(os.path.join("schemas", "interfaces", schemaFileName), "r") as schemaFile:
    schemaContent = json.load(schemaFile)

    schemaContentAsString = json.dumps(schemaContent)
    createdSchema = client.state.draft.schemas.create(schemaName, schemaFileName, schemaContentAsString, schemaDescription)
    print(createdSchema)

    This time you will find the reference code in configureLogicalDeviceModel.py, refer to the manageLiSchema() method that ensures the script runs idempotently.

    sysutilSchema = manageLiSchema(
    schemaName="sysutil-lischema",
    schemaFileName="sysutil-schema.json",
    description="Interface for anything capable of reading System Utilization data"
    )
    networkioSchema = manageLiSchema(
    schemaName="networkio-lischema",
    schemaFileName="networkio-schema.json",
    description="Interface for anything capable of reading Network I/O data"
    )

    Action

    Using the code snippets above register the two schemas (psutil-lischema and oshi-lischema) in your organization before proceeding to the next step

    More Information

    We are creating the logical interface schemas using exactly the same APIs that we used for the event type schemas in step 2.

  6. Create a logical interface

    When creating an logical interface you can optionally specify a meaningful alias for the interface, it is strongly encouragef that you always take advantage of this, it will make application development much easier than relying on the auto-generated UUID assigned to the interface.  The alias name carries the same restrictions that typeId and deviceId have, with one additional restriction to prevent collisions with the auto-generated UUIDs:

    • It must be 1 – 36 characters long
    • It can include alphanumeric, hypen, period, underscore characters.
    • It cannot be a 24 character hex string

     

    The following code snippet shows how to create the new logical interface using the schema we just created in the previous step:

    import wiotp.sdk.appplication

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    liName="sysutil-li",
    liAlias="sysutil",
    description="System Utilization Logical Interface",
    schemaId=createdSchema.id

    createdLI = client.state.draft.logicalInterfaces.create({
    "name": liName,
    "alias": liAlias,
    "description": description,
    "schemaId": schemaId
    })

    Refer again to configureLogicalDeviceModel.py, the manageLi() method wraps the code for setting up a logical interface to ensure the script can run idempotently.

    sysutilLI = manageLi(
    liName="sysutil-li",
    liAlias="sysutil",
    description="System Utilization Logical Interface",
    schemaId=sysutilSchema.id
    )
    networkioLI = manageLi(
    liName="networkio-li",
    liAlias="networkio",
    description="Network I/O Logical Interface",
    schemaId=networkioSchema.id
    )

    Action

    Using the code snippets above register two logical interfaces (psutil-li and oshi-li) in your organization before proceeding to the next step

    More Information

    To create a logical interface, we used the /draft/logicalinterfaces API:  Logical Interfaces API documentation.

     

  7. Add logical interfaces to the device types

    Just as we had to assign the physical interface to a device type, we must also assign the logical interface to a device type.  Although a device type can only have one physical interface, it can support multiple logical interfaces.  In this tutorial we are working with two device types: iotpsutil and iotoshi:

    • iotpsutil supports both of the logical interfaces we have created: sysutil-li and networkio-li
    • iotoshi only supports the sysutil-li logical interface, it is not capable of emitting events containing the necessary information to meet the requirements of the networkio interface.

     

    We add the logical interfaces to the device type as so:

    import wiotp.sdk.appplication

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    typeId="iotpsutil"

    deviceType = client.state.draft.deviceTypes[typeId]deviceType.logicalInterfaces.create(createdLI1)
    deviceType.logicalInterfaces.create(createdLI2)

    As before, we have wrapped the code in an idempotent wrapper manageDeviceType() in configureLogicalDeviceModel.py.

    psutilType = manageDeviceType("iotpsutil", [sysutilLI, networkioLI])
    oshiType = manageDeviceType("iotoshi", [sysutilLI])

    Action

    Using the code snippets above register two both logical interfaces on the iotpsutil device type, and only the sysutil-li logical interface to the iotoshi device type in your organization before proceeding to the next step

    More Information

    To add a logical interface to a device type, we used the /draft/device/types/{typeId}/logicalinterfaces API: Logical Interfaces API documentation.

  8. Define mappings for the logical interface

    We are nearly there, this is the last step required to complete the data management configuration for our two device types.

    import wiotp.sdk.appplication

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    typeId="iotpsutil"

    psutilMappings = [
    {
    "logicalInterfaceId": sysutilLI.id,
    "notificationStrategy": "on-state-change",
    "propertyMappings": {
    "psutil" : {
    "cpu" : "$event.cpu",
    "memory" : "$event.mem"
    }
    }
    },
    {
    "logicalInterfaceId": networkioLI.id,
    "notificationStrategy": "on-state-change",
    "propertyMappings": {
    "psutil" : {
    "network.inbound": "$event.network.down",
    "network.outbound": "$event.network.up"
    }
    }
    }
    ]
    typeId="iotpsutil"

    deviceType = client.state.draft.deviceTypes[typeId]for mapping in psutilMappings:
    deviceType.mappings.create(mapping)

    As usual, in configureLogicalDeviceModel.py we have wrapped this process into a method for idempotency: addMappings()

    addMappings(deviceType=psutilType, mappings=psutilMappings)
    addMappings(deviceType=oshiType, mappings=oshiMappings)

    Action

    Using the code snippets above complete the data management configuration for iotpsutil and iotoshi device types.

    More Information

    NotificationStrategy determines when applications subscribing to state changes will receive messages about device state changes.  By setting the strategy in the mapping we allow the same logical interface to be configured with a notification strategy per device type if required.  The notification strategy does not impact the internal processing of the device state in the platform, which is always evluated for each incoming event – it only controls the outbound notifications sent to applications that use MQTT to stream live changes to device state.

    To map events, we used the /draft/device/types/{typeId}/mappings API: Device Types API documentation.

     

  9. Activate the configuration

    We have now completed the three pieces of data management configuration: the physical interface, the logical interface, and how to translate from one to the other for our device types.  Before data management will start to do anything with this configuration it must be activated, and before we activate the configuration we will take advantage of two built-in capabilities of the platform:

    • differences() will inform us what has changed between the active configuration and the current draft configuration for any device type
    • validate() will alert us to any problems with the data management configuration for a device type.

     

    import wiotp.sdk.appplication

    options = wiotp.sdk.application.parseEnvVars()
    client = wiotp.sdk.application.ApplicationClient(options)

    print(client.state.draft.deviceTypes["iotpsutil"].differences())
    print(client.state.draft.deviceTypes["iotpsutil"].validate())

    When we are ready to activate the configuration we call activate():

    print(client.state.draft.deviceTypes["iotpsutil"].activate())

    Once again, refer to activateAndValidate.py, where you will see we have gated the differences, validate, and activate actions behind command line parameters.  To perform all three actions the script would be invoked as below:

    python activateAndValidate.py -d -v -a

    Action

    Compare, validate, and activate the data management configuration for all iotpsutil and iotoshi devices

    More Information

    To validate and activate your device type configuration, we used the /draft/device/types/{typeId} API: Device Types API documentation.

     

  10. Retrieve the state of the device

    Our data management configuration is now in effect, events coming into the platform from iotpsutil and iotoshi devices are now being used to generate and maintain a persistent state for each device, defined by the logical interfaces associated with those device types.

    The below code snippet shows how to access the current device state for the first 10 iotpsutil and iotoshi devices:

    for typeId in ["iotpsutil", "iotoshi"]:
    count = 0
    limit = 10
    for device in client.state.active.deviceTypes[typeId].devices:
    if count > limit:
    break
    print("%s:%s - %s" % (typeId, device.deviceId, device.states["sysutil"].state))
    count +=1

    The final step in activateAndValidate.py will do precisely this, if you run the script with no command line arguments it will simply print out the currrent state for your devices.

    Action

    Complete this tutorial by directly accessing the newly formed device state for your iotpsutil and iotoshi devices.  If you did not already connect devices using the reference samples as suggested at the start you will need to do so now if you want to see evidence that your configuration works.

    More Information

    You can retrieve the state of the device either using HTTP API, or subscribe via MQTT to recieve real-time notifications to device state changes (based on the notification strategy set up in the mappings).  Because we specified an alias name when creating the logical interfaces, we are able to use the aliases sysutil and networkio rather than needing to use the auto-generated UUID of our two logical interfaces.

    MQTT notifcations are available on the following topic:

    iot-2/type/${typeId}/id/${deviceId}/intf/${logicalInterfaceId}/evt/state

    The current state can always be retreived from the API:

    GET /device/types/{typeId}/devices/{deviceId}/state/{logicalInterfaceId}

     

  11. Next Steps

    Having learnt the basics of data management in IBM Watson IoT Platform why not try to accomplish one of the following tasks next:

    • Modify activateAndValidate so that the application subscribes to device state notifications and prints the updated state each time it changes
    • Configure embedded rules to trigger on device state changes
    • Scale up the number of connected devices using a Kubernetes cluster to run virtual devices using the deviceFactory sample and our psutil and oshi Docker images

Join The Discussion