Overview

Skill Level: Intermediate

Basic understanding of Node.js

This recipe will help you with regression testing your Watson Assistant (formerly Conversation) Chatbots. With this code and templates for easy intent classification or flow tests you will be able to ensure quality of your bots.

Ingredients

  • IBM Cloud Public Account
  • Git
  • Node.js installed on the machine running the tests

This asset has been tested with Watson Assistant API version 2018-02-16

Step-by-step

  1. Create chat bot with starter kit

    If you already have a bot you can skip the¬†chapters 1 ‚Äď 3 and get the essential information in chapter 4 “Write and execute an automated regression test”.

    Follow these steps to create a chat bot with starter kit:

    1.Login into IBM Cloud 

    2.Click on the menu in the upper left corner, next to the IBM Cloud Logo 

    3.Click on Web-Apps 

    Web-Apps

    4.In the section¬†“Start¬†from¬†the¬†Web”¬†click¬†on “Get¬†Started”¬†

    Start-from-the-Web5.Select Expressjs Watson Assistant Basic 

    Express.js-Watson-Assistant-Basic6.Select “Create¬†project”¬†

    Create-project7.In¬†the¬†upper¬†right¬†corner¬†click¬†on “Download¬†code”¬†

    Download-code

    In the Dashboard you will find a Watson Assistant instance and your browser should downloaded a zip file.

  2. Download and deploy the starterkit chatbot locally

    1.Extract the zip file

    2.Run “npm¬†i” in¬†the¬†extracted¬†folder¬†(node.js needed. Get it at https://nodejs.org/en/)

    npm-i

    3.Afterwards¬†run¬†“npm¬†run¬†start”¬†

    npm-run-start4.You¬†will¬†see¬†the¬†output¬†“creating¬†a¬†workspace…”¬†¬†when¬†you¬†run¬†the¬†script¬†the¬†first¬†time.¬†
    There will be no more success messages afterwards. 

    In¬†the¬†workspaces¬†overview¬†of¬†the¬†newly¬†created¬†Watson¬†Assistant¬†instance,¬†you¬†will now see¬†a¬†new¬†Workspace¬†called¬†“Car_Dashboard_Current”,¬†which¬†was¬†created¬†by¬†the¬†script¬†

    Car_Dashboard_Current

  3. Manually test the dialog flow

    With a running express server instance (after executing “npm¬†run start”) you can¬†open¬†http://localhost:3000¬†¬†
    This will show you a chat bot interface, which allows you to interact with the new create Car Dashboard workspace.

    WA-backend
    You can also login into the backend of the service and ensure that the service is working as intended. 

    WA-Workspace-backend

  4. Write and execute a automated regression test

    Download the test template code from the git repository.

    https://git.eu-de.bluemix.net/watmann/watson-assistant-regression-test

    You will find two test examples in the test folder.

    The file turn_off.js is a test for intents. With this template you can easily test the intent classification of your workspace. The second file, named turn_on_flow.js is a flow test, that tests a dialog flow including answers from your assistant.

    From our experience it is good to have at least a regression test for the intents right after or even before you create them. The problem of flow tests in early stages is that the answers from your dialog might change quite often and you have to correct the flow tests as well. So start with the intent regressiontests.

    There are 4 places where you have to enter your data:

    1.Tests Array:

    A test consists of several attributes. The question, which is the message/utterance that will be send to the /message endpoint of the Watson assistant API, the expected_intent, which defines the intent that should be recognized and type/value (expected_entity_type / expected_entity_value) of given entities that should be found in the utterance.

    test-attributes

     

    2.Flow-Flag:

    This flag determines if the tests should be handled as a flow test or as single requests, where the answers from your dialog won’t be evaluated.

    3.Min_Confidence:

    This variable determines the minimum confidence that you allow for your intent classification. If the returned confidence of an intent is below this threshold, the test will fail although the intent might be correct.

    4.env File:

    You have to create a file¬†in the root directory of the project, which is named “.env” (please don’t add any prefix in front of the dot). This file has to contain the credentials for your Watson Assistant instance. It has to look like this:

     

    ASSISTANT_URL= <Watson Assistant API URL>

    ASSISTANT_USERNAME= <Watson Assistant Username>

    ASSISTANT_PASSWORD= <Watson Assistant Password>

    ASSISTANT_WORKSPACEID= <Watson Assistent ‚Äď WorkspaceID>

    ASSISTANT_VERSION= <Watson Assistent API Version>

    The URL, username and password can be found in the credentials section of the service instance. Alternatively check the additional information section, at the end of this document, to find the credentials in the downloaded package. Click on the service in your dashboard and then on ‚ÄúService credentials‚ÄĚ on the left.

    WA-Service-Credentials

    The workspaceID can be found in the Watson Assistant Tooling. Click on the 3 dots in the right upper corner of a workspace and then on ‚Äúview details‚ÄĚ:

    WA-Workspace-view-details

    Now you are ready to start the tests. First run the command npm i in the root directory of the project. After all the dependencies are installed, you can run all the tests with ‚Äúnpm test‚ÄĚ or a single test with ‚Äúnpm test test/.js‚ÄĚ

  5. Additional information

    Another good source for information is the README.md, inside the downloaded Watson Assistant web starter.

    The web starter also contains your credentials for connecting to the Watson Assistant instance. You can find it in the localdev-config.json file, inside the downloaded Watson Assistant web starter, at the server folder.

    WA-Sample-Folder

  6. Authors

    Gerd Watnann
    Senior IT Architect

    Jan Roger Gerbig
    IT Specialist

    Kian Salem
    IT Specialist

3 comments on"Watson Assistant Automated Regression Test"

  1. Please feel free to comment, ask questions and provide feedback ūüôā

  2. This kind of functionality becomes really necessary when the Assistant is getting more complex and/or in maintenance mode.
    Apart from the Licensing Costs, do you have some pro’s and con’s on using this framework versus the Rational Test tooling for API’s that IBM also offers ?

    • Dear MJonker,

      first of all, please excuse the late response. Currently two of three authors (us) do not have hands on knowledge with Rational. The third one is in vacation. Maybe he can help you, when he returns.

      Anyway: I would focus on the following:
      1. Environment. If you have node js running on a machine, you can just use git, to get the current test system and keep it up to date. So no need of “real” software.
      2. Flow and intent test: Can you do this with rational? So we do not use question and answer, but couples of them. Sometimes one question relies on the answer of the previous one.

      If you are using Watson Assistant and creating your own frontend (and backend) for it, chances are good you are writing a node app anyway. So you can use the same environment for testing, als for development.

      I hope that helps for the moment
      Kind regards,
      Kian

Join The Discussion