The Blog

 

Artificial intelligence (AI) systems thrive on data, and usually the best source of this data is feedback from users interacting with your applications. Creating a virtuous feedback loop from your application to the AI services enabling it can help the system improve automatically without significant investment. Watson Discovery helps create this feedback loop through a set of new logging and events APIs, metrics generation, and Continuous Relevancy Training. With these capabilities, you can use Discovery to instrument and collect data from your application to see how users are interacting, including the queries they run and the results they select. This data is then automatically used by Discovery to generate metrics on usage, and perform Relevancy Training to improve the order of results returned.

figure 1

It all starts with collecting data. Discovery enables the data collection today through two ways.

  1. Automated data collection of queries and associated results. Any queries run against a Discovery environment get logged (unless opted out) and can be viewed from the /logs endpoint. These queries are valuable for analyzing user behavior and understanding opportunities to improve content or configuration.

  2. Collection of interaction data through the events API. Discovery provides an API that can be embedded in your application to track events that can be used as signals to understand usage and improve the service. This Events API currently supports tracking click events. Click events usually are associated with an action where a user selects a result from the list returned from Discovery, for example, selecting to see the full document text or clicking to expand a section of a document. These click events provide an indication of results that users believe to have some relevance, with the order of clicks also providing insight into relevance.

Taken together, this data helps provide a view into user behavior. These data points can then be used to generate metrics like those available through the Analytics tab in Discovery today. These metrics are an important way to track the performance of your application over time. They include:

  • Total queries run: See the volume of queries users are running over time to understand adoption and trends

  • Total queries with an associated event: Similar to a click-through rate, this can be an indicator of quality of results being returned and provide information on how the results might be improving

  • List of common query words: Used to see what kinds of queries users are doing so that you can focus efforts on improving content or training

Discovery also can use the feedback data collected to perform Continuous Relevancy Training to automatically improve results for certain types of queries. Continuous Relevancy Training learns from the collected data to rerank document results in the best possible order.

Continuous Relevancy Training uses state-of-the-art machine learning techniques to learn from fewer interactions. This means that different size organizations can use continuous learning without needing to have operations at global web scale. It can be used in across-enterprise use cases like customer service agent assist to provide agents with more relevant results from huge amounts of documentation to help them resolve customer issues quickly.

To use Continuous Relevancy Training, your data collection needs to be set up as previously mentioned and must meet the following requirements:

  • Collect at least 1000 queries with an associated click on a result in 30 days

  • Query Discovery at the environment level with the complete set of collections in that environment and track events at this level, too, so there is consistency between training and query

Continuous Relevancy Training combines the query and events logs to create a training set and then produces a model for reranking similar to one that can be created explicitly using Relevancy Training. To make the best use of Continuous Relevancy Training, there are a few considerations to keep in mind:

  • The query logs and events let you collect data on a larger scale, so it is not necessary to review individual events. Discovery can use the volume of data to find signals even if there are some ambiguous interactions. Instead, focus on getting a great data collection mechanism so that the events represent your users’ selections appropriately.

  • Labeling your data can help manage it going forward. For example, you can remove a set of data from test instances or automated processes that might not reflect real production users. Labeling is done through a customer-id header passed with each API request to Discovery.

  • You can test whether the model is improving results by comparing an environment-level query using natural_language_query to one using just the query parameter (natural_language_query only is supported for training). The boost in accuracy between using those two parameters is the benefit the new model gives you.

  • Explicit relevancy training (done through the training_data endpoint) is still valuable in cases where you are only querying a single collection. This form of training can be kept more curated while the continuous training can improve with behavior over time. Currently, data collected through feedback is not combined with training data explicitly provided.

Continuous Relevancy Training automates the feedback loop from your application. Interactions can automatically feed back to Discovery and be consumed for training. This provides a valuable improvement in relevance without as much investment in large, ongoing manual training. Get started creating your improvement loop today with Watson Discovery!

Note: Logging, Metrics and Continuous Relevancy Training are available in Watson Discovery for Salesforce today. Continuous Relevancy Training is available for only Advanced Plan instances with size S or greater.