Generate Core ML assets using IBM Maximo Visual Inspection

Introduction

Apple’s Core ML framework lets you deploy machine learning models onto Apple devices like iPhones and iPads. Core ML conversion generates files in the mlmodel format, which can be directly deployed into apps through Xcode on Apple devices for inference purposes.

This tutorial describes how to generate Core ML assets using IBM Maximo Visual Inspection. It also has examples of Xcode project for inference using the Core ML assets generated by Maximo Visual Inspection. I conclude by showing how to quickly mirror an iOS device on a Mac for presentations or demos.

Prerequisites

  • The ability to install tools and packages.
  • The ability to run programs like Python and Xcode
  • A copy of IBM Maximo Visual Inspection
  • Completed installation of Xcode 11.3+ and Python 3.6+
  • Completed installation of the coremltools package

Estimated time

Users with prior experience installing Python packages and running Python and Xcode will need an estimated 3 hours to complete this tutorial. This assumes that the Core ML formatted deep learning models are already available to be downloaded from IBM Maximo Visual Inspection application. Time taken to train deep learning models on Maximo Visual Inspection application depends on the data provided and model type selected.

Steps

In machine learning, the training step is where we create a model using the input data sets and an algorithm. Inference is the step when we use the trained model to predict on new test data.

Using Core ML tools, you can convert a classification or object detection model into Core ML format. Alternatively, you can generate Core ML assets by using Maximo Visual Inspection. When training a model for classification you can specify that you want to generate Core ML assets. From training a model for object detection (YOLO v3), by default a Core ML asset is generated. These Core ML assets can be downloaded and directly deployed onto mobile devices for further inference.

Use IBM Maximo Visual Inspection to generate Core ML assets

Maximo Visual Inspection supports the conversion of GoogleNet (for classification) and Tiny YOLO V2/YOLO V3 models (for object detection) into the Core ML format. After training an appropriate model, a Core ML asset is generated. It can be downloaded and used for inference purposes.

  1. On the Maximo Visual Inspection Data sets page, upload a valid dataset zip file with images or create a new dataset and add images to it.
  2. Add categories of images and assign images to each category for image classification. For object classification models, label the objects in the images by drawing bounding boxes around the object and assigning them to the right object name.
  3. On the Data sets page, click on Train Model to train the YOLO v3 model.
  4. In the Optimize model using area of the Train Model page, select Enable Core ML.

For details on how to complete the above steps, visit the IBM Knowledge Center’s Maximo Visual Inspection pages.

alt

Perform prediction using a Python script

Inference of image classification model

This example uses a model that is trained for different classes of birds. In the code below, googlenet.mlmodel is the file that’s downloaded and renamed from Maximo Visual Inspection’s Trained Model’s Model Assets tile.

The following python program uses the downloaded model to do inferences. It takes an image as input and provides the classification result of the image.

import coremltools  

# Load an image using PIL  
from PIL import Image  
coreml_model =  coremltools.models.MLModel('googlenet.mlmodel')

name = "larus-bird.jpg"
with Image.open(name) as img:
    img224 = img.resize((224,224))
    rslt = coreml_model.predict({'data': img224})
    sorted_x = sorted(rslt['prob'].items(), key=lambda kv: kv[1], reverse=True)
    print(sorted_x[:3])

Sample result if a bird image is used as input:

[('Larus', 0.9999508857727051), ('Butorides', 4.908662958769128e-05), ('Corvus', 1.4055167030235793e-09)]

Sample result if a dog image is used as input:

[('_negative_', 0.9998596906661987), ('Haliaeetus', 7.920272037154064e-05), ('Parus', 2.137967931048479e-05)]

Inference using Xcode project

Download the example code for Xcode. Replace model in the Xcode with the model downloaded from Maximo Visual Inspection and renamed googlenet.mlmodel.

alt

Negative inference result for image classification

The model is trained with birds categories and a cup is given as input for inference. Result comes out as _negative_ showing a category that’s not present in the input data set.

alt

Positive inference result for image classification

When a bird image is given as input, the model correctly predicts the category.

alt

YOLO v3 object detection model training

This model is trained in Maximo Visual Inspection with an Arduino boards data set to identify different board types in an image. On the data set page, when Train Model is selected, you can see options for training type YOLO v3 and corresponding optimization. For YOLO v3, generating Core ML asset is the default behavior. alt

The image below shows the Xcode project changed to include the downloaded asset. The downloaded asset has been downloaded from Maximo Visual Inspection and renamed YOLOv3.mlmodel. Test images have been added to images.bundle folder in the project.

alt

Inference using a new image with Arduino dashboards

Download the code for the Xcode Arduino project.

alt

Inference result

#filename,predicted-name,xmin,ymin,xmax,ymax,coreml-confidence
both_boards.png,arduino_mega,178,88,1111,544,0.9992
both_boards.png,arduino_uno,272,748,1023,1246,0.9901

The results above show the result of filename given, found object name from the labeled objects, and coordinates that show the localization of the objects.

Mirroring an iPhone onto a Mac for inference for Xcode programs

In situations where a physical device is selected as the target device instead of the Xcode simulator, and users want to see the device directly on their Mac laptop screen, follow these steps to see it there:

  1. Connect the iOS device to your Mac.
  2. Open QuickTime Player.
  3. Navigate to File > New Movie Recording.
  4. Click the down arrow to the right of the record button.
  5. Under Camera, select the name of your iOS device.
  6. Click on the red record button to begin recording video from your iPhone.
  7. Click on the button again to stop recording.

Summary

In this tutorial, you learned:

  • How to get Core ML assets from image classification and object detection models using IBM Maximo Visual Inspection.
  • How to use Core ML assets generated for inferencing purposes using command line and Xcode projects.
  • How to mirror an iPhone onto a Mac for inference for Xcode programming.

You can gain knowledge on how to run deep learning models directly on their devices. On-device inferencing removes any need for a network connection and helps in keeping user’s data private and app responsive. Different data sets can be trained on Maximo Visual Inspection and the resulting Core ML models can be deployed onto mobile devices like an iPhone or an iPod.

Check out the links in the Resources section to continue your learning.