The Case for Visual Inspection for Quality Control in Manufacturing
Rigorous quality control techniques have become an essential differentiator in the manufacturing and production industry today. The automobile industry is an example where these techniques are increasingly important as they pertain to compliance to passenger safety and regulatory compliance standards. The manufacturing process of a car involves the assembly of hundreds of, if not more, parts. Many parts are stamped using metal forming procedures. One of the main components that must pass rigorous quality controls are steel plates. In this tutorial you will learn how IBM Maximo Visual Inspection can be used to detect defects on steel plate surfaces as part of the automated manufacturing process to maximize the efficiency and effectiveness of the quality inspection processes.
IBM Maximo Visual Inspection is a visual recognition platform with built-in deep learning models and an intuitive interface that can be used to build models for classification and object detection of images and video streams. It includes an intuitive toolkit that allows subject matter experts to create deep learning vision models, without deep learning or coding expertise. It includes the most popular deep learning infrastructures and their dependencies, as well as facilitating and accelerating deployment and increasing team productivity. By combining IBM Maximo Visual Inspection with accelerated IBM Power Systems, companies can quickly deploy a fully optimized and supported platform with the best performance. In order to complete this tutorial you will need access to an instance of IBM Maximo Visual Inspection.
You will also need to download the Supervised Learning for Industrial Optical Inspection data set that has been made publicly available by The Heidelberg Collaboratory for Image Processing (HCI) or a sub-set of it that is referenced in this tutorial.
This tutorial assumes that you are familiar with Maximo Visual Inspection. Start with the Getting started with IBM Maximo Visual Inspection learning path if you are unfamiliar with Maximo Visual Inspection.
Understand the data set
The data set referenced in this tutorial is a sub-set of the data set above that has been made available by HCI. Download the sample data sets. This data set contains labelled training and test images to detect defects on steel surfaces. The data set is composed of different images, showing the real surface of steel plates, with and without defects. For the purpose of this tutorial, the subset of the data set used to detect defects is categorized as:
Hole defects: This type of defect occurs when holes are drilled in the surface of the metal plate by accident during the manufacturing process.
Line defects: This kind of defect occurs when a slit machine acts on a wrong surface.
Scratch defect: This defect is a result of scratches that can occur on the surface of metal plates in the production line.
Temperature defect: The process of steel solidification is very specific and sometimes, certain areas of the steel sheets solidify under local conditions that cause a temperature defect on the surface.
The goal of this use case is to reduce the cost of industrial quality control through visual inspection which is used to identify manufacturing defects and take correcting actions through automated process and decision management. For this specific use case, the three data sets that have been provided are a subset of the HCI data set and include:
5x4WithoutNondefect: This data set contains a few images from each defect so it is the ideal data to manually label and build the preliminary model with which we would apply the auto labeling.
12x4WithoutNonDefect_Autolabel: This folder contains a larger number of images. In this data, the preliminary model is going to be applied in order to automatically categorize the images and then build the model that is going to be used for the test.
4x4With20Nondefect_ForTEST: This is a small sample of 16 defect images and 20 non-defect images. The model is going to be tested with these images.
You will now build a deep learning visual inspection model to classify industrial defects on steel surfaces and then proceed to test and evaluate the model.
Create a new data set in Maximo Visual Inspection
- Log into Maximo Visual Inspection.
- Select the Data sets tab in the navigation bar on Maximo Visual Inspection to open the Data Sets page. There are several ways to create a new data set. For this tutorial, create a new, empty data set by clicking the Create a new data set button. Set the new data set name to
Steel Defectand click Create to create the new empty data set.
- To add an image to the data set, click the newly created Steel Defect data set which opens the Data Set view. Import training data images by dragging and dropping the training image files and then clicking the Import files button to import these images. Import the 5x4WithoutNondefect data set provided. It is small enough to be labelled manually.
- Click the Add files button to import the training image data files into the new data set in Maximo Visual Inspection. Once all images have been imported, they will appear in the view of the Steel Defect Data Set.
Important: You cannot navigate away from the Maximo Visual Inspection page or refresh until the upload completes, however you can navigate to different pages within Maximo Visual Inspection during the upload.
Label objects in the training data set
The next step is to label objects in the training images that were imported. In order to train an object detection model with Maximo Visual Inspection, you must have a minimum of five labelled images for each object. Now proceed to labelling five images for each of the objects: Hole, Line, Scratch and Temperature. These objects and their corresponding labels represent manufacturing defects on steel surfaces that need to be inspected.
- Select the images from your data set and click Label Objects.
- Add 4 object labels for Hole, Line, Scratch and Temperature by clicking the Add Object button from the Objects left menu bar item and entering the object label in the Add Objects pop-up dialog box.
- Label the objects in the imported data set images. Open each image in the data area by clicking the image. This opens a thumb-nail preview; click the Label Objects button in the preview. Select the correct object label from the menu on the right and then choose Box or Polygon from the menu at the bottom left, depending on the shape you want to draw around each object. Boxes are faster to label and train, but less accurate. Only Detectron models support polygons.
Repeat the labelling steps for each image in the set of imported training images.
- After all objects are labeled in all of the image, click Done editing.
Train the object detection model
With all the object labels that are identified in your data set, you can now train your deep learning model. To train a model, complete the following steps:
- From the data set view, click the Train model button.
- Fill out the fields on the Train Data set page, ensuring that you select Object Detection for type of training. There are different training options that give different results as we will see in the conclusion of this guide.
- Click Train.
- Depending on the size of the training data set and the options selected, the training time and accuracy varies. You can monitor the progress as the object detection model is being trained.
Training the steel defect object detection model
To deploy the trained model, complete the following steps:
- Click Models from the menu.
- Select the model you created in the previous section and click Deploy.
- Specify a name for the model and click Deploy. The Deployed Models page is displayed, and the model is deployed when the status column displays Ready.
- Double-click the deployed model to get the API endpoint and test other videos or images against the model. For information about using the API see Vision Service API documentation.
Note: Each Tiny YOLO V2, YOLO V3, Detectron, Single Shot Detector (SSD), Structured segment network (SSN), or custom deployed model takes one GPU. The GPU group is listed as ‘-‘, which indicates that this model uses a full GPU and does not share the resource with any other deployed models. Multiple faster R-CNN and GoogLeNet models are deployed to a single GPU. IBM Maximo Visual Inspection uses packing to deploy the models. This means that the model is deployed to the GPU that has the most models deployed on it if there is sufficient memory available on the GPU. The GPU group can be used to determine which deployed models share a GPU resource. To free up a GPU, all deployed models in a GPU group must be deleted (undeployed).
Auto-label a larger dataset
Now that the manually labelled object detection model has been built and deployed, the option to auto-label a larger data set is available. Auto labelling provides an autonomous method to label the data set. Auto label the larger data set using this option and then a new model which will have better accuracy with the larger data set that is used for training.
- Click the Data set tab from the menu.
- Create a new data set like you did before in the Create a new data set in Maximo Visual Inspection section above by importing training images from the archive 12x4WithoutNonDefect_Autolabel. Notice that the images you imported in this dataset are not labelled.
- Click on the Auto label option and choose the deployed model with which you want to auto label the images.
- You will see a confirmation notification once the labelling is complete. Notice that the images this dataset are now labelled.
- Now train a new model with the auto labelled data set as done above in the Training the steel defect object detection model section.
- Deploy the model as done in the Training the steel defect object detection model section.
Test the object detection model
Now proceed to test the object detection models created. The model created with a larger set should have a better accuracy. When testing models for accuracy, Maximo Visual Inspection has a Confidence threshold parameter can be specified which indicates the accuracy with which the model detect objects in images. It is a good indicator of the quality of the model and a data point that MVI provides when testing your models for accuracy.
To test the models that were created, navigate to the Models tab and to the desired deployed model and click on it to open the model.
Navigate to the Test Model section at the bottom of this page. Upload the test images by dragging and dropping them in the Drop image or video here section and click the Import button. Test images have been provided in the 4x4With20Nondefect_ForTEST archive.
Maximo Visual Inspection will import test images, detect objects, and return the results of testing. Any of the 4 classes of objects that correspond to steel surface defects will be automatically detected.
Results and Conclusions
In the steps above, you chose the default options to train and build an object detection model. In addition to the data set used, these choices you make can impact the accuracy and speed of the trained object detection model. Compare and analyze the results of three different models built and tested to determine which model provides the most accurate results and performs the best which helps make the most informed decision that could speed up and improve the quality control process.
- Fast R-CNN: This convolutional neural network is highly accurate for locating object bounding boxes in both images and videos. It generates a TensorRT asset by default.
After applying this model to the 4x4With20Nondefect_ForTEST data set, the results were much better than the model generated with the default options. The original model was able to successfully detect 14 out of 16 defects in the images in the data set and this model was able to detect all 16.
- Detectron: This approach is a priori the best for object detection and segmentation of small objects. It generates accurate but large models.
After applying this model to 4x4With20Nondefect_ForTEST data set, the results were a little worse than expected. The original data set model was able to successfully find 8 out of 16 defects in the images and the data set model to which auto label was applied was able to successfully detect 11. The lower accuracy rate is probably because this model is more oriented to object segmentation.
- YOLO: This is optimized for speed but can be less accurate. It generates smaller models that work well for FPGA and embedded devices.
After applying this model to 4x4With20Nondefect_ForTEST data set, the results were a little worse than expected. The original data set model was able to successfully find 10 out of 16 defects in the images and the data set model to which auto label was applied was able to successfully detect 12.
IBM Maximo Visual Inspection can help detect defects on steel plate surfaces as part of the automated manufacturing process to maximize the efficiency and effectiveness of the quality inspection processes. We look forward to hearing how you use it in your work.