Blog Post

Watson Discovery versus watsonx Discovery: A comparison

Learn which product is better for your needs


LikeSave



Generative AI has arrived, creating a once-in-a-generation opportunity for businesses. According to Goldman Sachs generative AI could raise global GDP by 7% within 10 years. One of the leading reasons why a significant breakthrough has been achieved in generative AI is development in deep learning that involves using neural networks to learn from data.

Today, AI models are no longer constrained by the amount of data to learn from. Generative AI models can learn from billions of examples, which has made the models more accurate and realistic. This learning has also been made possible with the transformer architecture.

Traditionally, recurrent neural networks (RNNs) and convolutional neural networks (CNNs) were unable to efficiently process long sequences of data, a common characteristic of any natural language. In a transformer architecture, the self-attention mechanisms allow the transformer to assign weights to each element in the sequence based on its relevance to the other elements in the sequence. These models provide the ability to process input sequences in parallel as compared to the sequential processing done by RNNs. This makes the models more powerful and relevant.

New algorithms with the transformer architecture have evolved that, for example, can be trained to create large language models (LLMs) that could be used for several applications and use cases. The key advantage of using LLMs is getting more accurate results without training on large amounts of data or just doing some amount of fine tuning. LLMs fall into a category called foundation models.

The IBM watsonx platform uses foundation models and is an AI and data platform with three core components:

It also has a set of AI assistants that are designed to help you scale and accelerate the impact of AI with trusted data across your business. It helps train, tune, and deploy AI models, scale data where it resides, and design trustworthy AI workflows.

IBM Watson Discovery

IBM Watson Discovery is focused on creating intelligent business processes by automating the discovery of information and insights to empower the enterprise's knowledge workers. With businesses going digital and the democratisation of social media platforms and mobile usage, data volumes, structured, semi-structured, and unstructured, are increasing. The need for insight engine technology to surface facts or content is all the more critical.

Watson Discovery is the complete solution for document and language intelligence that accelerates high-value insight processes across the enterprise. Some of the key features include:

  • Extract data from human-read documents with Smart Document Understanding (SDU). SDU can visually teach Watson Discovery how to interpret document structures and components. This enables you to quickly get accurate insights even if the data is stored in semi-structured formats like tables.

  • Enrich text with advanced out-of-the box natural language processing (NLP). You can use the included models that understand keywords, entities, and concepts.

  • Teach Discovery to improve with input from business users. Advanced NLP customization is used to teach Discovery the domain language of the business by using label examples and custom dictionaries. You can extract similarly patterned information from your data to find insights at scale.

  • Analyze data at-scale to find answers, insights, and patterns. Find answers or a fact from a long passage, or generate FAQs. Using in combination with the watsonx Assistant Search skill allows you to have seamless conversations without the "dead end" in dialog flows and without training for all possible customer questions

IBM watsonx Discovery

IBM watsonx Discovery is an add-on to watsonx Assistant that helps resolve informational tasks through a generative AI assistant. It is an IBM-packaged Elasticsearch to leverage semantic, federated, and vector search over business-specific content.

Customer experience and digital labor markets are key areas that are using generative AI. LLMs are being used for automating tasks, guiding users on how to self-serve, or using conversational search to answer business topics. Conversational search uses retrieval-augmented generation (RAG), an AI framework that retrieves data from external sources of knowledge to improve the quality of responses.

For conversational search with watsonx Discovery, enterprise-quality RAG requires the following three components:

  • A conversational AI platform that is a chat interface that can scale across multiple subjects and provide out-of-the-box system integrations as well as provide a framework that reduces development time.

  • A retrieval system that performs a semantic, federated, and vector search across data sources.

  • Enterprise-ready and trusted LLMs for supporting LLM use cases like generation and summarization of content from the retrieval system. It should provide for a seamless chat to generate and summarize personalized answers based on the business content returned by the retrieval system. The models are fine-tuned to provide a good chat experience by knowing when to be brief or verbose and when to return an "I don’t know" answer.

IBM is a leader in the Gartner Magic Quadrant for all of the previous key features except for the strong semantic search requirements by using vector search as provided by Elastic. With vectors, you can overcome the limitations of traditional keyword-based search by using machine learning models to capture the meaning of words and phrases in context, rather than relying solely on lexical analysis and matching of individual query terms. Vector search can return more relevant results that match your needs, even if the exact terms aren't present in the document.

Coupled with watsonx Assistant, watsonx Discovery is used for semantic search, and LLMs (running on watsonx.ai) are used to generate content-grounded conversational answers to customer and employee questions.

High-level archicture

Key differences between Watson Discovery and watsonx Discovery

The following points highlight the key differences between watson Discovery (intelligent document processing) and watsonx Discovery (semantic search).

  • Watson Discovery is good for document understanding and processing, whereas watsonx Discovery is good for conversational semantic search.

  • Watson Discovery is a stand-alone product that creates intelligent business processes. watsonx Discovery is an add-on to watsonx Assistant that helps resolve informational tasks through a generative AI assistant.

  • Watson Discovery looks to empower knowledge workers to create intelligent business processes by automating the discovery of information and insights. Watsonx Discovery empowers customers and employees to resolve their informational tasks by relying on a generative AI assistant that can handle all of their topics.

  • Watson Discovery uses domain-specific NLP capabilities to process documents with intelligent document processing, including deep enrichments of documents. Watsonx Discovery uses LLMs for conversational search.

  • Watson Discovery is embedded in business processes and leverages domain-specific NLP. Watsonx Discovery is highly scalable with simple configuration with watsonx Assistant.

  • Watson Discovery is available on IBM Cloud and Cloud Pak for Data. Watsonx Discovery is available on Cloud Pak for Data as of December 2023.

  • Watson Discovery supports multiple languages. Watsonx Discovery currently only supports English.