Overview

Skill Level: Beginner

basic NodeJS skills

A common problem with email support from customers is that 100s-->1000s of emails come into 1st line support who have to use their judgement to analyse the content and direct to the right support team. This process can be greatly improved. Let's see how.

Ingredients

  • Bluemix Account - Sign up for a trial account if you don't already have one.
  • Download and install NodeJS (including npm) for your platform
  • Create an NLC (Natural Language Classifier) service

Step-by-step

  1. Download and install

    Download and install NodeJS to your machine (node v0.8.0 or newer)

    Check that it works okay from a terminal command-line:

    • node –version
    • npm –version

     

     

    Setup a couple of GMail email accounts

    • First one is to read incoming emails from
    • Second is the one to send emails to

     

    For a full demonstration of this, you could have multiple “Second” emails to demonstrate sending to different support groups.

     

    Github download of sample code

    • npm install

    Modules used:

    • cfenv
    • emailjs
    • express
    • imap
    • mailparser
    • watson-developer-cloud

    The app has a sample UI web page to invoke the NodeJS code for demonstration purposes, this is written in AngularJS v1 (for simplicity), it does not do anything amazing and should be very easy to understand and extend (if you want to upgrade it to AngularJS v2, go ahead and do so and share it back with me).

    Read the following, but remember to modify the emailpoc.js and common.js files with YOUR credentials!

  2. Application Architecture & POSTMAN testing

    This application is built using NodeJS and is currently built to just execute once, for demonstration purposes.  You can easily extend (and I encourage you to do so) to either have an event driven mechansim to be aware of new emails arriving in the Inbox or worst-case use a polling mechanism to request if their are any new emails (just don’t do it ever 5milliseconds or GMail might not like you)

    Email_Arch

    Whilst we are using GMail IMAP/SMTP to read/send the emails, this could easily be switched out to connect to an Enterprise email system – the concept is the same, you will just have to modify the details based on the connectivity you need.

    I’ve also built a very simple UI web page that invokes the Express REST API calls to perform each of the individual component elements:

    • getEmails
    • classify
    • sendToGroup

     

    You can also just as easily use POSTMAN to call the APIs directly passing the content required and see the returned responses.

     

    We can call /getEmails and this will respond with the email parsed and extracted as shown below:

    demo_postman_01

    We can extract the “text” and place this into the BODY to POST to the /classify API:

    demo_postman_02

    This will respond with the following:

    demo_postman_03

    This data was provided by passing the “text” to the NLC and the “top_class” and “classes” are passed back as-is.

    This is where the /classify API nodeJS code can be extended to have which group email address to use depending upon the “confidence” level and the “class_name”, or just from the “top_class” value.

    We can test the /sendToGroup API code by using the “text” and “subject” data from the initial call to /getEmails earlier.  We will determine the “from” email address as explained above.  For this example, I’ll use a default address.

    demo_postman_04

    When the /sendToGroup API is called, it will then send the email to the defined email address.  You can then check in GMail to see the email.

     

  3. Setup the Natural Language Classifier

    As defined in the ingredients section, it is assumed that you have an IBM Bluemix account. Sign in to your account and select [Catalog] [Watson]. The service that we are interested is highlighted below:

    nlc_new_01

    Select the [Natural Language Classifier] service to create a new instance:

    nlc_new_02

     

    Once created, select the NLC service:

    nlc_service_02

     

    Now, we need to know the Service Credentials to insert into the emailpoc.js file (so we know which NLC service to connect to).  In order to do that we need to select “Service Credentials” and click on “View Credentials”:

    nlc_service_03-1

    This is 2 of the 3 pieces of data that we need to enter into the emailpoc.js file (lines 24/25).

    If we click “Manage” and go back to the previous screen, select the [Access the beta toolkit], this will open in a new window.

    nlc_service_04

    The “Classifier ID” is the last value that we need to enter into the emailpoc.js file (line 28).

     

     

    I have created a sample CSV file that contains some initial ground truth data (support_data_train.csv):

    nlc_csv_train_data_01

    The format is simple.  Enter the Utterance with a comma and then the category name.  I’ve included a limited sample here, but feel free to add a lot more to obtain a greater confidence scoring.

    We shall import this into the NLC like so, select “Training data” and the [import] icon.  This is open a dialog for you to select the .csv file.

    nlc_service_05

    Once imported you will see the values like so:

    nlc_service_06

     

    We can perform some tests directly, just to make sure all is okay:

    nlc_service_07

     

     

  4. Walkthrough and breakdown of code

    As the application is written in NodeJS, you can run this locally on your own machine or you can “cf push” it up to the IBM Cloud Bluemix platform and run it from there – either way, you’ll get the same result.  I prefer to run it locally to start off with and to debug until I’m happy.

    As shown above, you can manually call the REST APIs from POSTMAN or you can use the sample UI demo.

    In a terminal window, navigate to the project folder and execute

    $node emailpoc.js

    You will get an output that will tell you what PORT has been assigned (usually :6013).

    In a web browser, you can navigate to:  http://localhost:6013/

    The index.html page is the container page that via the common.js file content will then display the mainpage view and invoke the controller to display the mainpage.html page to you:

    index_01 common_01mainpage_01

    Here is the rendered web page:

    demo_ui_01

    If you take a look at the code for the mainpage.html you’ll see that the ng-click actions will invoke functions within the mainpage.js controller code.

    By pressing the [Get Emails] button you will call the getEmails() function, as you can see this is the code that is executed:

    mainpagejs_03

    This then calls the same API that we called with POSTMAN earlier on.  This then executes the code within the emailpoc.js node application:

    getEmails_01getEmails_02getEmails_03

    As you can see the return of that API in emailpoc.js is returned to the mainpage.js controller function.

    We then split out the “text”, “from” and “subject” values and put them into $scope variables that are then populated and displayed in the web page:

    demo_ui_02

     

    If you now press the [Classify] button, this will invoke the function within the mainpage.js controller code:mainpagejs_02

    This will then call the emailpoc.js API:

    classify_01

    As you can see, this code is executing the Watson Developer Cloud – Natural Language Classifier library and passes the “text” to the NLC service.  The response is then just passed back as-is to the calling mainpage.js controller function.

    The returned value is then parsed and the values put into $scope variable and refreshed in the web page:

    demo_ui_03

    As you can see we are just using this for DEBUGGING information at the moment.

     

    Now time for you to complete this sample:

    The [Send To Group] button is not actually coded yet.  But, you can guess what it is going to do.  I’ll leave that piece of work for you to complete.

    In essense, the mainpage.html page tells you the name of the function in the ng-click action to name the function in the mainpage.js controller.  You can then pretty much copy & paste the same content from the .classify code (as we’re doing a POST call).  The difference is that you will want to put different code into the “data” object that you are going to POST.  Hint: Take a look at the POSTMAN call we made earlier to see what the emailpoc.js “/v1/sendToGroup” API call is expecting to receive.

    The next logical step would be for you to extend the emailpoc.js “/v1/classify” API call to parse the response data (extract the top_class and classes content) and insert code to determine which email group the classified email should be sent to.  Then you could pass this email address back to the calling mainpage.js controller function.  This value can then be passed when the [Send To Group] button is pressed.

     

     

Join The Discussion