Explore text classification with Watson NLP

With IBM Watson Libraries, IBM introduced a common library for artificial intelligence (AI) runtimes (for serving the model) and AI libraries (like Natural Language Processing, Document Understanding, Translation, and Trust). IBM Watson Libraries brings everything under one umbrella for consistency and ease of development and deployment. This tutorial walks you through the steps of training a model to classify text in consumer complaints by using the watson_nlp library from IBM Watson NLP.

The watson_nlp library is available on IBM Watson Studio as a runtime library so that you can directly use it for model training, evaluation, and prediction. The following figure shows the Watson NLP architecture.

IBM Watson NLP is a standard embeddable AI library that is designed to tie together the pieces of IBM Natural Language Processing. It provides a standard base natural language processing layer along with a single integrated roadmap, a common architecture, and a common code stack designed for widespread adoption across IBM products.

Text Classification is usually manually processed by humans to gather groups of qualitative data. Having the ability to automatically gather and process larger data sets of text through customer feedback, comments, or an entire article that is written on your product is a strong tool to gain insight into the most common emotional responses in a group of people or a block of text.

IBM Watson NLP now provides the ability to automatically classify the input text into one or more predetermined sets of labels.

This tutorial explains the fundamentals of IBM Watson NLP and walks you through the process of training and evaluating the models to perform text classification.

Prerequisites

To follow the steps in this tutorial, you need:

• An IBMid
• A Watson Studio project
• A Python notebook
• Your environment set up

Before working through the tutorial, you should have an understanding of IBM Watson Studio and Jupyter Notebooks.

Estimated time

It should take you approximately 1 hour to complete this tutorial.

Steps

The steps in this tutorial use an example of a Consumer complaint database to walk you through the process.

Step 1. Data processing and exploratory data analysis

1. Download the data set. The data set is collected from the Consumer Financial complaint database. To use the data, the data set is normalized by removing rows that do not have a value of consumer complaints. This data set contains 999285 consumer complaints with the date received, submitted via, products, sub-products, and company information.

2. Downsample the data set to reduce model training time and quick analysis.

   complaint_df = complaint_df.sample(frac=0.02)

3. Look at all of the product groups that are available in the data set because these are the classes that the classifier should predict from a given complaint text.

   

4. Filter on the Product categories with a relevant number of samples and remove any other product category from further analysis because many classification algorithms work best if the training samples are equally split across the classes. If the data is unbalanced, algorithms might decide to favor classes with many samples to achieve an overall good result.

   train_test_df = complaint_df[(complaint_df['Product'] == 'Credit reporting, credit repair services, or other personal consumer reports') | \
                                (complaint_df['Product'] == 'Debt collection') | \
                                (complaint_df['Product'] == 'Mortgage') | \
                                (complaint_df['Product'] == 'Credit card or prepaid card') | \
                                (complaint_df['Product'] == 'Checking or savings account')
                               ]
   

5. Split the data into training and test data (ratio: 80/20).

   # 80% training data
   train_orig_df = train_test_df.groupby('Product').sample(frac=0.8, random_state=6)
   print("Training data:")
   print("Number of training samples: {}".format(len(train_orig_df)))
   print("Samples by product group:\n{}".format(train_orig_df['Product'].value_counts()))
   
   # 20% test data
   test_orig_df = train_test_df.drop(train_orig_df.index)
   print("\nTest data:")
   print("Number of test samples: {}".format(len(test_orig_df)))
   print("Samples by product group:\n{}".format(test_orig_df['Product'].value_counts()))
   
   # re-index after sampling
   train_orig_df = train_orig_df.reset_index(drop=True)
   test_orig_df = test_orig_df.reset_index(drop=True)
   

   You have created two DataFrames, one for the training and one for the test data. The data is still in its original format. Now, you must convert the data into a format that is usable by the Watson NLP classification algorithms. This can be either JSON or CSV formats.

6. Create the data in a JSON format. The training and test data is written to files.

   

Step 2. Model building

Step 2a. Train a TF-IDF SVM classification model with Watson NLP

SVM is an established classification approach. Watson NLP includes an SVM algorithm that exploits the SnapML libraries for faster training. The algorithm utilizes TF-IDF embeddings.

The TF-IDF SVM classifier workflow depends on the syntax block. So, start by loading the syntax model.

1. Load the syntax model and the USE embeddings because the SVM classifier block depends on the syntax block.

   # Syntax Model
   syntax_model = watson_nlp.load(watson_nlp.download('syntax_izumo_en_stock'))
   

2. Create data streams using several utility methods because classification blocks expect the training data to be in data streams.

   training_data_file = train_file
   
   # Create datastream from training data
   data_stream_resolver = DataStreamResolver(target_stream_type=list, expected_keys={'text': str, 'labels': list})
   training_data = data_stream_resolver.as_data_stream(training_data_file)
   
   # Create Syntax stream
   text_stream, labels_stream = training_data[0], training_data[1]
   syntax_stream = syntax_model.stream(text_stream)
   

3. Train the classifier.

   tfidf_classification_model = TFidfSvm.train(training_data=training_data,
                       syntax_model=syntax_model, 
                       tfidf_svm_epochs=1,
                       multi_label=True
                      )
   

Step 2b. Train a USE SVM classification model with Watson NLP

This algorithm utilizes Universal Sentence Encoder (USE) embeddings that encode word-level semantics into a vector space.

1. The syntax model has already been loaded in the previous step so the next step is to load a USE (Universal Sentence Encoder) model

   use_embedding_model = watson_nlp.download_and_load('embedding_use_en_stock')
   

2. Train the USE SVM classifier

   embeddings_classification_model = UseSvm.train(training_data=training_data,
                       syntax_model=syntax_model, 
                       use_embedding_model=use_embedding_model, 
                       use_svm_epochs=1,
                       multi_label=True
                      )
   

Step 2c. Train a GloVe CNN model with Watson NLP

CNN is a simple convolutional network architecture that is built for multiclass and multilabel text classification on short texts. The GloVe CNN workflow simplifies the training process of Glove embedding + CNN.

1. Start by loading the GloVe embedding model.

   glove_embedding_model = watson_nlp.download_and_load('embedding_glove_en_stock')
   

2. Train the CNN model with GloVe embedding.

   cnn_model = GloveCNN.train(
                          training_data=training_data,
                          syntax_model=syntax_model, 
                          glove_embedding_model=glove_embedding_model,
                          cnn_epochs=1,
                                     )
   

Step 2d. Train an ensemble classification model with Watson NLP

The ensemble model combines three classification models:

• CNN with GloVe embeddings
• SVM with TF-IDF features
• SVM with USE (Universal Sentence Encoder) features

It computes the weighted mean of classification predictions using confidence scores. You use the default weights, which can be fine-tuned in subsequent steps.

The ensemble workflow is easy to use, and the model performance can be much better than individual algorithms, depending on the syntax model and the GloVe and USE embeddings. They are passed with the file that contains the training data.

1. Train the ensemble classifier.

   Note: To restrict the time, we limited the epochs to train the CNN classifier to 10. This is an optional attribute. If it is not specified, the default is 30 epochs.

   ensemble_model = GenericEnsemble.train(train_stream, 
                                                syntax_model, 
                                             base_classifiers_params=[
                                                   TFidfSvm.TrainParams(syntax_model=syntax_model, tfidf_svm_epochs=10),
                                                   UseSvm.TrainParams(syntax_model=syntax_model, use_embedding_model= use_embedding_model,
                                                                     use_svm_epochs=10),
                                                   GloveCNN.TrainParams(syntax_model=syntax_model,
                                                                        glove_embedding_model=glove_embedding_model,
                                                                        cnn_epochs=10,
                                                                     )],
                                             weights=[1,1,1])
   

Step 2e. Store and load classification models (optional)

You can save a model as a project asset. model.as_file_like_object() creates a .zip file, which is provided as a BytesIO object that is stored in the project.

1. Save both models in your project.

   project.save_data('tfidf_classification_model', data=tfidf_classification_model.as_file_like_object(), overwrite=True)
   

   project.save_data('ensemble_model', data=ensemble_model.as_file_like_object(), overwrite=True)
   

Step 3. Model evaluation

Now you are able to run the trained models on new data. You run the models on the test data so that the results can also be used for model evaluation. For illustration purposes, the data is used in the original format that you began with because the format of the new complaints that you receive might also be in that format.

1. Create a helper method to run both models on a single complaint and return the predicted product groups of both models. Notice that the SVM model requires you to run the syntax model on the input texts first.

   def predict_product(text):
      tfidf_svm_preds = tfidf_classification_model.run(text)
   
      predicted_tfidf_svm = tfidf_svm_preds.to_dict()["classes"][0]["class_name"]
   
      ensemble_preds = ensemble_model.run(text)
      predicted_ensemble = ensemble_preds.to_dict()["classes"][0]["class_name"]
      return (predicted_tfidf_svm, predicted_ensemble)
   

2. Run the models on the complete test data.

   

3. Plot a confusion matrix for the classifiers.

   

You see that the precision, recall, and f1-measure for some classes is much lower than for others. The reason might be that it is difficult to differentiate between some classes.

Overall, the ensemble model performs at par with the SVM model. However, the SVM model had a significantly shorter training time.

Conclusion

This tutorial showed you how to use the Watson NLP library as well as how to quickly and easily train and run different text classifiers using Watson NLP.

Next steps

You can also run this Jupyter Notebook, which demonstrates another use case for classifying hotel reviews.

In this notebook, you learn how to prepare your data, train a custom SVM model and an ensemble model, save the model and score data and compare model quality on test data.