IBM Developer

Tutorial

Build a RAG system with LangChain, Python, and watsonx

Learn how retrieval-augmented generation (RAG) combines semantic search and large language models to retrieve relevant context and generate accurate, grounded responses

By Nicholas Moss, Erika Russi

On this page

In this tutorial, you use LangChain to build a step-by-step retrieval-augmented generation (RAG) example in Python.

RAG is a technique in natural language processing (NLP) that combines information retrieval and generative models to produce more accurate, relevant, and contextually aware responses.

For this use case, you set up a RAG system for IBM Think 2024. IBM Think 2024 is a conference where IBM announces new products, technologies, and partnerships. You fetch content from several ibm.com websites, which makes up a knowledge base that you provide to an LLM as context to answer questions about Think 2024.

Watch this video to see how to use LangChain RAG for this use case:

LangChain RAG tutorial video thumbnail
Video will open in new tab or window

More about RAG and LangChain

In traditional language generation tasks, large language models (LLMs) like OpenAI's GPT-3.5 (Generative Pre-trained Transformer) or IBM's Granite Models are used to construct responses based on an input prompt. However, these models might struggle to produce responses that are contextually relevant, factually accurate, or up to date. The models might not know the latest information on IBM Think 2024.

RAG applications address this limitation by incorporating a retrieval step before response generation. During retrieval, vector search can be used to identify contextually pertinent information, such as relevant information or documents from a large corpus of text, typically stored in a vector database. Finally, an LLM is used to generate a response based on the retrieved context.

LangChain is a powerful open-source framework that facilitates the development of applications using LLMs for various NLP tasks. In the context of RAG, LangChain plays a critical role by combining the strengths of retrieval-based methods and generative models to enhance the capabilities of NLP systems.

Prerequisites

You need an IBM Cloud account to create a watsonx.ai project.

Steps

Step 1. Set up your environment

While you can choose from several tools, this tutorial walks you through how to set up an IBM account to use a Jupyter Notebook. Jupyter Notebooks are widely used in data science to combine code, text, images, and data visualizations to formulate a well-formed analysis.

  1. Log in to watsonx.ai using your IBM Cloud account.

  2. Create a watsonx.ai project.

  3. Create a Jupyter Notebook.

This step opens a Notebook environment where you can copy the code from this tutorial to implement a RAG application for Think 2024. Alternatively, you can download this notebook to your local system and upload it to your watsonx.ai project as an asset. This Jupyter Notebook is available on GitHub.

Step 2. Set up the watsonx.ai Runtime service and API key

  1. Create a watsonx.ai Runtime service instance (choose the Lite plan, which is a free instance).
  2. Generate an API Key.
  3. Associate the watsonx.ai Runtime service to the project that you created in watsonx.ai.

Step 3. Install and import relevant libraries and set up credentials

You need a few libraries for this tutorial. Import the ones below. If they are not installed, run a quick pip install.

#installations
%pip install langchain
%pip install langchain_chroma
%pip install langchain-community
%pip install -U langchain_ibm
%pip install unstructured
%pip install ibm-watsonx-ai
%pip install nltk

Import the relevant libraries:

#imports
import os
import getpass

from ibm_watsonx_ai.foundation_models.utils.enums import ModelTypes
from ibm_watsonx_ai.metanames import GenTextParamsMetaNames as GenParams
from ibm_watsonx_ai.foundation_models.utils.enums import EmbeddingTypes

from langchain_ibm import WatsonxEmbeddings, WatsonxLLM
from langchain_chroma import Chroma

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

from langchain_community.document_loaders import UnstructuredURLLoader

from langchain_text_splitters import RecursiveCharacterTextSplitter

import nltk
nltk.download('averaged_perceptron_tagger_eng')

Set up your credentials and input your API Key:

credentials = {
    "url": "https://us-south.ml.cloud.ibm.com",
    "apikey": getpass.getpass("Please enter your WML api key (hit enter): ")
}

Set up your project_id as part of your environment variables or input it:

try:
    project_id = os.environ["PROJECT_ID"]
except KeyError:
    project_id = input("Please enter your project_id (hit enter): ")

Step 4. Index the URLs to create the knowledge base

Index Think 2024-specific articles from URLs to create a knowledge base as a vectorstore. The content from these URLs serves as the data source and context for this exercise. The context is then provided to an LLM to answer questions about the Think 2024 conference.

The first step to building vector embeddings is to clean and process the raw data set. This might involve the removal of noise and the standardization of the text. For this example, no cleaning is required because the text is already cleaned and standardized.

First, establish URLS_DICTIONARY. URLS_DICTIONARY is a dict that helps map the URLs from which you extract content. Also set up a name for your collection: askibm_think_2024.

URLS_DICTIONARY = {
    "ibm.com_events_think_faq.html": "https://www.ibm.com/events/think/faq",
    "events_think_agenda.html": "https://www.ibm.com/events/think/agenda",
    "products_watsonx_ai.html": "https://www.ibm.com/products/watsonx-ai",
    "products_watsonx_ai_foundation_models.html": "https://www.ibm.com/products/watsonx-ai/foundation-models",
    "watsonx_pricing.html": "https://www.ibm.com/watsonx/pricing",
    "watsonx.html": "https://www.ibm.com/watsonx",
    "products_watsonx_data.html": "https://www.ibm.com/products/watsonx-data",
    "products_watsonx_assistant.html": "https://www.ibm.com/products/watsonx-assistant",
    "products_watsonx_code_assistant.html": "https://www.ibm.com/products/watsonx-code-assistant",
    "products_watsonx_orchestrate.html": "https://www.ibm.com/products/watsonx-orchestrate",
    "products_watsonx_governance.html": "https://www.ibm.com/products/watsonx-governance",
    "granite_code_models_open_source.html": "https://research.ibm.com/blog/granite-code-models-open-source",
    "red_hat_enterprise_linux_ai.html": "https://www.redhat.com/en/about/press-releases/red-hat-delivers-accessible-open-source-generative-ai-innovation-red-hat-enterprise-linux-ai",
    "model_choice.html": "https://www.ibm.com/blog/announcement/enterprise-grade-model-choices/",
    "democratizing.html": "https://www.ibm.com/blog/announcement/democratizing-large-language-model-development-with-instructlab-support-in-watsonx-ai/",
    "ibm_consulting_expands_ai.html": "https://newsroom.ibm.com/Blog-IBM-Consulting-Expands-Capabilities-to-Help-Enterprises-Scale-AI",
    "ibm_data_product_hub.html": "https://www.ibm.com/products/data-product-hub",
    "ibm_price_performance_data.html": "https://www.ibm.com/blog/announcement/delivering-superior-price-performance-and-enhanced-data-management-for-ai-with-ibm-watsonx-data/",
    "ibm_bi_adoption.html": "https://www.ibm.com/blog/a-new-era-in-bi-overcoming-low-adoption-to-make-smart-decisions-accessible-for-all/",
    "watsonx_code_assistant_for_z.html": "https://www.ibm.com/blog/announcement/ibm-watsonx-code-assistant-for-z-accelerate-the-application-lifecycle-with-generative-ai-and-automation/",
    "code_assistant_for_java.html": "https://www.ibm.com/blog/announcement/watsonx-code-assistant-java/",
    "code_assistant_for_orchestrate.html": "https://www.ibm.com/blog/announcement/watsonx-orchestrate-ai-z-assistant/",
    "accelerating_gen_ai.html": "https://newsroom.ibm.com/Blog-How-IBM-Cloud-is-Accelerating-Business-Outcomes-with-Gen-AI",
    "watsonx_open_source.html": "https://newsroom.ibm.com/2024-05-21-IBM-Unveils-Next-Chapter-of-watsonx-with-Open-Source,-Product-Ecosystem-Innovations-to-Drive-Enterprise-AI-at-Scale",
    "ibm_concert.html": "https://www.ibm.com/products/concert",
    "ibm_consulting_advantage_news.html": "https://newsroom.ibm.com/2024-01-17-IBM-Introduces-IBM-Consulting-Advantage,-an-AI-Services-Platform-and-Library-of-Assistants-to-Empower-Consultants",
    "ibm_consulting_advantage_info.html": "https://www.ibm.com/consulting/info/ibm-consulting-advantage"
}
COLLECTION_NAME = "askibm_think_2024"

Next, load the documents using the LangChain UnstructuredURLLoader for the list of URLs. Print a sample document at the end to see how it is loaded.

documents = []

for url in list(URLS_DICTIONARY.values()):
    loader = UnstructuredURLLoader(urls=[url])
    data = loader.load()
    documents += data

#show sample document
documents[0]

Output:

Document(page_content="View session catalog\n\nEvent information\n\nThink 2024 will be held in Boston, MA.\xa0 It will open with an exclusive, partner-only event on 20 May, filled with inspiration, networking, knowledge-sharing and business value for IBM’s global ecosystem of partners. Then, on 21–23 May, we will bring together senior business and technology leaders from across industries for two and a half days of exploration, discussion and innovation.\n\nThink 2024 will host senior business and technology leaders from across industries. Content will be geared toward C-level, line of business and senior IT leaders.\n\nThink 2024 programming will be held at the Boston Convention & Exhibition Center (BCEC), with some activities at the Omni Boston Hotel at the Seaport.\n\nAt IBM, we are committed to sustainability and environmentally responsible event planning. We are proud to partner with two distinguished venues, each known for their exemplary sustainable practices. Our event will take place at the Boston Convention & Exhibition Center and Omni Seaport in Boston. These venues share our dedication to reducing environmental impact, conserving resources and promoting eco-friendly practices. We are excited to collaborate with them to ensure that Think 2024 and our special events not only deliver an exceptional experience, but also align with our commitment to a greener, more sustainable future.\n\nRegistration\n\nYes, the Think 2024 event in Boston will be a fee-based event. Standard pricing is USD 1,899, effective 16 January through 23 May 2024.\n\nRequests for cancellation of a paid Think pass must be made in writing to\xa0IBMThink@gpj.com. Requests received by 19 April 2024, 11:59 PM ET will receive a full refund minus an administrative fee of USD 150. After midnight ET on Saturday, 20 April 2024, no refunds will be issued. Non-attendance does not constitute cancellation. If you have extenuating circumstances, please reach out with an e-mail to\xa0IBMThink@gpj.com.\n\nSubstitutions for purchased passes will be allowed for attendees from the same company, and an administrative fee of USD 150 will be applied to the new registration.\n\nSubstitutions for complimentary pass holders are not permitted. Please contact your IBM account representative or the Think Guest Services Team at\xa0IBMThink@gpj.com.\n\nIndividuals must be over the age of 21 to participate in conference activities. Infants and young children, even under the supervision or care of an adult, may not attend conference activities at the Boston Convention and Exhibition Center and Omni Boston Hotel. This includes all activities, sessions, meal functions, meeting and exposition space, and the use of event transportation if applicable. There may be stated exceptions where individuals under the age of 21 are allowed with a purchased guest conference badge.\n\nIf you are having a problem logging into your IBMid, or if you need to create an IBMid, please contact the\xa0IBMid help desk.\n\nPlease contact Think Guest Services via between 9 AM and 6 PM ET.\n\nUS/Canada: +1-866-382-7150\n\nInternational: +1-650-416-1881\n\nEmail:\xa0IBMThink@gpj.com\n\nHotel and Travel\n\nThe official Think hotel blocks are closed. We may still have rooms available for new requests. To inquire about a new hotel booking, please email IBMThink@gpj.com and provide your arrival and departure information. Think Guest services will respond within 24 hours. For hotel changes or cancellations, please contact the hotel directly after you receive your hotel confirmation email with confirmation number.\n\nCancellation or changes to hotel reservations are the sole responsibility of the attendee. Cancellations within 72 hours of the reservation check-in date will result in a late cancellation fee of 1 night’s room and tax. If you do not check in on your scheduled arrival date, the remainder of your reservation will be canceled.\n\nYour hotel loyalty information can be added at the time of booking or with the hotel directly.\n\nFor hotel changes or cancellations, please contact the hotel directly and provide your hotel confirmation number which is included with your hotel confirmation email.\n\nIf you need to arrive early or extend your stay outside of the event dates and dates you booked with Think Guest Services, please contract the hotel directly and provide them with you hotel confirmation number. Rates will be based on the hotel's availability and are not guaranteed at the conference rate.\n\nIf you did not request at the time of booking your reservation with Think Guest Services, please contact the hotel directly. Please note that all requests are based on hotel availability and are not guaranteed.\n\nFor IBM clients and Business Partners requesting a visa invitation letter, please contact the Think Customer Service team via email at\xa0IBMThink@gpj.com and include the following information:\n\nName on passport\n\nCompany name\n\nAddress\n\nCity\n\nCountry\n\nPlease note that an invitation letter will be provided to registered attendees only. Letters cannot be provided for guests.\n\nAt the event\n\nYes, IBM is dedicated to providing a safe, respectful, comfortable, and harassment-free environment for all participants at IBM events.\n\nLearn more (link resides outside ibm.com)\xa0\xa0about IBM's Code of Conduct at events.\n\nNo, unfortunately conference passes are limited and reserved for invited and registered attendees.\n\nThe conference attire is business casual. Comfortable shoes are recommended. Weather conditions can vary, so please review the local weather prior to your departure for the conference.\n\nShuttle service will not be provided from conference hotels to the Boston Convention and Exhibition Center (BCEC). The recommended options for transportation would be through your choice of Uber/Lyft/Taxi or a short walk. Hotels are within a 3 mile radius to the BCEC.\n\nGround transportation is not provided from/to the airport. As attendees will arrive and depart at different times, the recommended options for transportation would be through your choice of Uber/Lyft/Taxi or a scheduled private pickup.\n\nNo, unfortunately attendance to the Wednesday evening “Blue Block Party” special event is limited and reserved for invited and registered attendees.\n\nTo self-park at the BCEC ($25 and $50 for oversized vehicles), from Summer Street, turn onto East Side Drive, drive past the valet area, and continue straight along the side of the building. At the end of the building, make a right and go down the ramp. At the bottom of the ramp, turn left and you will see the entrance to the South Parking lot in front of you. Accessible parking spaces are also located in this lot, closest to the building (the same parking rates apply).\n\nView nearby parking lots and garage information here\xa0(link resides outside ibm.com).\n\nDiversity and Inclusion\n\nIBM is committed to creating a respectful, friendly and inclusive experience for all participants. Diversity, equity and inclusion are core components of IBM's culture. IBM's conferences include elements such as accessibility, prayer/meditation rooms, and parent/mother rooms to help foster a deeper sense of inclusion and a welcoming experience for all participants.\n\nIf you have an accessibility-related need, please let us know in your registration profile or email Guest Services directly at IBMThink@gpj.com, ideally at least two to three weeks prior to the conference. Once we receive your request, one of our ambassadors will contact you.\n\nWhen we know, you’ll know\n\nGet updates on speakers, sessions and essential conference information, delivered right to your inbox.\n\nSubscribe", metadata={'source': 'https://www.ibm.com/events/think/faq'})

Based on the sample document, there is white space and new line characters to remove. Clean up the document and add metadata, including an ID number and the source of the content.

doc_id = 0
for doc in documents:
    doc.page_content = " ".join(doc.page_content.split()) # remove white space

    doc.metadata["id"] = doc_id #make a document id and add it to the document metadata

    print(doc.metadata)
    doc_id += 1

Output:

    {'source': 'https://www.ibm.com/events/think/faq', 'id': 0}
    {'source': 'https://www.ibm.com/events/think/agenda', 'id': 1}
    {'source': 'https://www.ibm.com/products/watsonx-ai', 'id': 2}
    {'source': 'https://www.ibm.com/products/watsonx-ai/foundation-models', 'id': 3}
    {'source': 'https://www.ibm.com/watsonx/pricing', 'id': 4}
    {'source': 'https://www.ibm.com/watsonx', 'id': 5}
    {'source': 'https://www.ibm.com/products/watsonx-data', 'id': 6}
    {'source': 'https://www.ibm.com/products/watsonx-assistant', 'id': 7}
    {'source': 'https://www.ibm.com/products/watsonx-code-assistant', 'id': 8}
    {'source': 'https://www.ibm.com/products/watsonx-orchestrate', 'id': 9}
    {'source': 'https://www.ibm.com/products/watsonx-governance', 'id': 10}
    {'source': 'https://research.ibm.com/blog/granite-code-models-open-source', 'id': 11}
    {'source': 'https://www.redhat.com/en/about/press-releases/red-hat-delivers-accessible-open-source-generative-ai-innovation-red-hat-enterprise-linux-ai', 'id': 12}
    {'source': 'https://www.ibm.com/blog/announcement/enterprise-grade-model-choices/', 'id': 13}
    {'source': 'https://www.ibm.com/blog/announcement/democratizing-large-language-model-development-with-instructlab-support-in-watsonx-ai/', 'id': 14}
    {'source': 'https://newsroom.ibm.com/Blog-IBM-Consulting-Expands-Capabilities-to-Help-Enterprises-Scale-AI', 'id': 15}
    {'source': 'https://www.ibm.com/products/data-product-hub', 'id': 16}
    {'source': 'https://www.ibm.com/blog/announcement/delivering-superior-price-performance-and-enhanced-data-management-for-ai-with-ibm-watsonx-data/', 'id': 17}
    {'source': 'https://www.ibm.com/blog/a-new-era-in-bi-overcoming-low-adoption-to-make-smart-decisions-accessible-for-all/', 'id': 18}
    {'source': 'https://www.ibm.com/blog/announcement/ibm-watsonx-code-assistant-for-z-accelerate-the-application-lifecycle-with-generative-ai-and-automation/', 'id': 19}
    {'source': 'https://www.ibm.com/blog/announcement/watsonx-code-assistant-java/', 'id': 20}
    {'source': 'https://www.ibm.com/blog/announcement/watsonx-orchestrate-ai-z-assistant/', 'id': 21}
    {'source': 'https://newsroom.ibm.com/Blog-How-IBM-Cloud-is-Accelerating-Business-Outcomes-with-Gen-AI', 'id': 22}
    {'source': 'https://newsroom.ibm.com/2024-05-21-IBM-Unveils-Next-Chapter-of-watsonx-with-Open-Source,-Product-Ecosystem-Innovations-to-Drive-Enterprise-AI-at-Scale', 'id': 23}
    {'source': 'https://www.ibm.com/products/concert', 'id': 24}
    {'source': 'https://newsroom.ibm.com/2024-01-17-IBM-Introduces-IBM-Consulting-Advantage,-an-AI-Services-Platform-and-Library-of-Assistants-to-Empower-Consultants', 'id': 25}
    {'source': 'https://www.ibm.com/consulting/info/ibm-consulting-advantage', 'id': 26}

See how the sample document looks after the cleanup:

documents[0]

Output:

Document(page_content="View session catalog Event information Think 2024 will be held in Boston, MA. It will open with an exclusive, partner-only event on 20 May, filled with inspiration, networking, knowledge-sharing and business value for IBM’s global ecosystem of partners. Then, on 21–23 May, we will bring together senior business and technology leaders from across industries for two and a half days of exploration, discussion and innovation. Think 2024 will host senior business and technology leaders from across industries. Content will be geared toward C-level, line of business and senior IT leaders. Think 2024 programming will be held at the Boston Convention & Exhibition Center (BCEC), with some activities at the Omni Boston Hotel at the Seaport. At IBM, we are committed to sustainability and environmentally responsible event planning. We are proud to partner with two distinguished venues, each known for their exemplary sustainable practices. Our event will take place at the Boston Convention & Exhibition Center and Omni Seaport in Boston. These venues share our dedication to reducing environmental impact, conserving resources and promoting eco-friendly practices. We are excited to collaborate with them to ensure that Think 2024 and our special events not only deliver an exceptional experience, but also align with our commitment to a greener, more sustainable future. Registration Yes, the Think 2024 event in Boston will be a fee-based event. Standard pricing is USD 1,899, effective 16 January through 23 May 2024. Requests for cancellation of a paid Think pass must be made in writing to IBMThink@gpj.com. Requests received by 19 April 2024, 11:59 PM ET will receive a full refund minus an administrative fee of USD 150. After midnight ET on Saturday, 20 April 2024, no refunds will be issued. Non-attendance does not constitute cancellation. If you have extenuating circumstances, please reach out with an e-mail to IBMThink@gpj.com. Substitutions for purchased passes will be allowed for attendees from the same company, and an administrative fee of USD 150 will be applied to the new registration. Substitutions for complimentary pass holders are not permitted. Please contact your IBM account representative or the Think Guest Services Team at IBMThink@gpj.com. Individuals must be over the age of 21 to participate in conference activities. Infants and young children, even under the supervision or care of an adult, may not attend conference activities at the Boston Convention and Exhibition Center and Omni Boston Hotel. This includes all activities, sessions, meal functions, meeting and exposition space, and the use of event transportation if applicable. There may be stated exceptions where individuals under the age of 21 are allowed with a purchased guest conference badge. If you are having a problem logging into your IBMid, or if you need to create an IBMid, please contact the IBMid help desk. Please contact Think Guest Services via between 9 AM and 6 PM ET. US/Canada: +1-866-382-7150 International: +1-650-416-1881 Email: IBMThink@gpj.com Hotel and Travel The official Think hotel blocks are closed. We may still have rooms available for new requests. To inquire about a new hotel booking, please email IBMThink@gpj.com and provide your arrival and departure information. Think Guest services will respond within 24 hours. For hotel changes or cancellations, please contact the hotel directly after you receive your hotel confirmation email with confirmation number. Cancellation or changes to hotel reservations are the sole responsibility of the attendee. Cancellations within 72 hours of the reservation check-in date will result in a late cancellation fee of 1 night’s room and tax. If you do not check in on your scheduled arrival date, the remainder of your reservation will be canceled. Your hotel loyalty information can be added at the time of booking or with the hotel directly. For hotel changes or cancellations, please contact the hotel directly and provide your hotel confirmation number which is included with your hotel confirmation email. If you need to arrive early or extend your stay outside of the event dates and dates you booked with Think Guest Services, please contract the hotel directly and provide them with you hotel confirmation number. Rates will be based on the hotel's availability and are not guaranteed at the conference rate. If you did not request at the time of booking your reservation with Think Guest Services, please contact the hotel directly. Please note that all requests are based on hotel availability and are not guaranteed. For IBM clients and Business Partners requesting a visa invitation letter, please contact the Think Customer Service team via email at IBMThink@gpj.com and include the following information: Name on passport Company name Address City Country Please note that an invitation letter will be provided to registered attendees only. Letters cannot be provided for guests. At the event Yes, IBM is dedicated to providing a safe, respectful, comfortable, and harassment-free environment for all participants at IBM events. Learn more (link resides outside ibm.com) about IBM's Code of Conduct at events. No, unfortunately conference passes are limited and reserved for invited and registered attendees. The conference attire is business casual. Comfortable shoes are recommended. Weather conditions can vary, so please review the local weather prior to your departure for the conference. Shuttle service will not be provided from conference hotels to the Boston Convention and Exhibition Center (BCEC). The recommended options for transportation would be through your choice of Uber/Lyft/Taxi or a short walk. Hotels are within a 3 mile radius to the BCEC. Ground transportation is not provided from/to the airport. As attendees will arrive and depart at different times, the recommended options for transportation would be through your choice of Uber/Lyft/Taxi or a scheduled private pickup. No, unfortunately attendance to the Wednesday evening “Blue Block Party” special event is limited and reserved for invited and registered attendees. To self-park at the BCEC ($25 and $50 for oversized vehicles), from Summer Street, turn onto East Side Drive, drive past the valet area, and continue straight along the side of the building. At the end of the building, make a right and go down the ramp. At the bottom of the ramp, turn left and you will see the entrance to the South Parking lot in front of you. Accessible parking spaces are also located in this lot, closest to the building (the same parking rates apply). View nearby parking lots and garage information here (link resides outside ibm.com). Diversity and Inclusion IBM is committed to creating a respectful, friendly and inclusive experience for all participants. Diversity, equity and inclusion are core components of IBM's culture. IBM's conferences include elements such as accessibility, prayer/meditation rooms, and parent/mother rooms to help foster a deeper sense of inclusion and a welcoming experience for all participants. If you have an accessibility-related need, please let us know in your registration profile or email Guest Services directly at IBMThink@gpj.com, ideally at least two to three weeks prior to the conference. Once we receive your request, one of our ambassadors will contact you. When we know, you’ll know Get updates on speakers, sessions and essential conference information, delivered right to your inbox. Subscribe", metadata={'source': 'https://www.ibm.com/events/think/faq', 'id': 0})

We need to split up our text into smaller, more manageable pieces known as "chunks." LangChain's RecursiveCharacterTextSplitter takes a large text and splits it based on a specified chunk size using a predefined set of characters. In order, the default characters are:

  • "\n\n" - two new line characters
  • "\n" - one new line character
  • " " - a space
  • "" - an empty character

The process starts by attempting to split the text using the first character, "\n\n." If the resulting chunks are still too large, it moves to the next character, "\n," and tries splitting again. This continues with each character in the set until the chunks are smaller than the specified chunk size. Because all the "\n\n" and "\n" characters were removed during the cleanup step, the RecursiveCharacterTextSplitter begins at the " " (space) character.

A chunk size of 512 was selected after experimenting with a chunk size of 1000. When the chunks were that large, the model received too much context for question-answering, which led to inaccurate responses. You can experiment with chunk size further.

text_splitter = RecursiveCharacterTextSplitter(chunk_size=512, chunk_overlap=0)
docs = text_splitter.split_documents(documents)

Next, choose an embedding model to train on the Think 2024 data set. The trained embedding model generates embeddings for each data point in the data set. For text data, popular open-source embedding models include Word2Vec, GloVe, FastText, or pre-trained transformer-based models like BERT or RoBERTa. OpenAI embeddings can also be used by leveraging the OpenAI embeddings API endpoint and the langchain_openai package with an openai_api_key, however, there is a cost associated with this usage.

Because embedding models are large, vector embeddings often demand significant computational resources. You can lower the costs linked to embedding vectors, while preserving performance and accuracy, by using WatsonxEmbeddings. This tutorial uses the IBM embeddings model Slate, an encoder-only (RoBERTa-based) model that is fast and effective for many NLP tasks.

embeddings = WatsonxEmbeddings(
    model_id=EmbeddingTypes.IBM_SLATE_30M_ENG.value,
    url=credentials["url"],
    apikey=credentials["apikey"],
    project_id=project_id,
    )

Load the content into a local instance of a vector database, using Chroma.

vectorstore = Chroma.from_documents(documents=docs, embedding=embeddings)

Run a quick search of the vector database to test it. Using similarity_search_with_score returns the documents and the distance score of the query. The returned distance score is Euclidean distance, so a lower score is better.

prompt = "What is IBM concert?"
search = vectorstore.similarity_search_with_score(prompt)
search

Output:

    [(Document(page_content='and solving problems before they happen. Concert will initially focus on helping application owners, SREs and IT leaders gain insights about, pre-empt and more quickly address issues around application risk and compliance management. Read this blog to learn more about IBM Concert. IBM expands ecosystem access to watsonx, adds third-party models IBM continues to foster a strong ecosystem of partners to offer clients choice and flexibility through bringing third-party models onto watsonx, enabling leading', metadata={'id': 23, 'source': 'https://newsroom.ibm.com/2024-05-21-IBM-Unveils-Next-Chapter-of-watsonx-with-Open-Source,-Product-Ecosystem-Innovations-to-Drive-Enterprise-AI-at-Scale'}),
    0.3013177514076233),
    (Document(page_content='can easily move to and operate across multi-cloud and hybrid cloud environments. Now, at THINK, IBM is continuing to advance its state-of-the-art in automation portfolio by previewing a new generative AI-powered tool called IBM Concert, which will be generally available in June 2024. IBM Concert will serve as the ‘nerve center’ of an enterprise’s technology and operations. Powered by AI from watsonx, IBM Concert will provide generative AI-driven insights across clients’ portfolios of applications to', metadata={'id': 23, 'source': 'https://newsroom.ibm.com/2024-05-21-IBM-Unveils-Next-Chapter-of-watsonx-with-Open-Source,-Product-Ecosystem-Innovations-to-Drive-Enterprise-AI-at-Scale'}),
      0.3116500675678253),
     (Document(page_content='IBM Concert Simplify and optimize your app management and technology operations with generative AI-driven insights Book a live demo Prioritize and act on the most significant risks to your business applications Explore risk management with IBM Concert Keep applications continuously compliant, even as they evolve Explore compliance management with IBM Concert What is IBM Concert? Get the overview IBM® Concert® puts you in control, so you can simplify and optimize your operations to focus on continuously', metadata={'id': 24, 'source': 'https://www.ibm.com/products/concert'}),
      0.31172141432762146),
     (Document(page_content='documentation for details. Context length is expressed in tokens. The IBM statements regarding its plans, directions and intent are subject to change or withdrawal without notice at its sole discretion. See Pricing for more details. Unless otherwise specified under Software pricing, all features, capabilities and potential updates refer exclusively to SaaS. IBM makes no representation that SaaS and software features and capabilities are the same.', metadata={'id': 3, 'source': 'https://www.ibm.com/products/watsonx-ai/foundation-models'}),
      0.43464165925979614)]

Now ask "Where is Think 2024?"

prompt = "Where is Think 2024?"
search = vectorstore.similarity_search_with_score(prompt)
search

Output:

    [(Document(page_content='keynotes from Think 2024 and explore related content in the Think Hub. Watch on-demand Stay in the know. Catch up on all the action from Think 2024 and watch event replays. Watch on-demand', metadata={'id': 1, 'source': 'https://www.ibm.com/events/think/agenda'}),
      0.20512934029102325),
     (Document(page_content='View session catalog Event information Think 2024 will be held in Boston, MA. It will open with an exclusive, partner-only event on 20 May, filled with inspiration, networking, knowledge-sharing and business value for IBM’s global ecosystem of partners. Then, on 21–23 May, we will bring together senior business and technology leaders from across industries for two and a half days of exploration, discussion and innovation. Think 2024 will host senior business and technology leaders from across industries.', metadata={'id': 0, 'source': 'https://www.ibm.com/events/think/faq'}),
      0.3093597888946533),
     (Document(page_content='Think 2024 Think 2024 2024-05-20 Monday: IBM Partner Plus Day Programming for Business Partners 2024-05-21 Tuesday: Think programming kicks off Event registration begins. Scaling your business with AI and hybrid cloud. Join IBM and other industry experts for engaging sessions and demos in the Meeting Center. Unleash the transformative power of AI and automation. Chef-selected lunch items and connections with your peers. Full lunch menu in the Think mobile app. Scale productivity with watsonx assistants.', metadata={'id': 1, 'source': 'https://www.ibm.com/events/think/agenda'}),
      0.32775259017944336),
     (Document(page_content='Seaport in Boston. These venues share our dedication to reducing environmental impact, conserving resources and promoting eco-friendly practices. We are excited to collaborate with them to ensure that Think 2024 and our special events not only deliver an exceptional experience, but also align with our commitment to a greener, more sustainable future. Registration Yes, the Think 2024 event in Boston will be a fee-based event. Standard pricing is USD 1,899, effective 16 January through 23 May 2024. Requests', metadata={'id': 0, 'source': 'https://www.ibm.com/events/think/faq'}),
      0.3494681119918823)]

Step 5. Set up a retriever

We'll set up our vector store as a retriever. The retrieved information from the vector store serves as additional context or knowledge that can be used by a generative model.

retriever = vectorstore.as_retriever()

Step 6. Generate a response with a generative model

Finally, generate a response. The generative model (like GPT-4 or IBM Granite) uses the retrieved information to produce a more accurate and contextually relevant response to questions about Think 2024.

First, establish which LLM to use to generate the response. For this tutorial, use IBM's Granite 13B Chat model.

model_id = "ibm/granite-3-8b-instruct"

The model parameters reference lists the available parameters. This tutorial experimented with various model parameters, including Temperature, Top P, and Top K. See the model parameters documentation for more information on model parameters and what they mean.

parameters = {
    GenParams.DECODING_METHOD: 'greedy',
    GenParams.TEMPERATURE: 2,
    GenParams.TOP_P: 0,
    GenParams.TOP_K: 100,
    GenParams.MIN_NEW_TOKENS: 10,
    GenParams.MAX_NEW_TOKENS: 512,
    GenParams.REPETITION_PENALTY:1.2,
    GenParams.STOP_SEQUENCES:['B)','\n'],
    GenParams.RETURN_OPTIONS: {'input_tokens': True,'generated_tokens': True, 'token_logprobs': True, 'token_ranks': True, }
}

Next, instantiate the LLM.

llm = WatsonxLLM(
    model_id=model_id,
    url=credentials.get("url"),
    apikey=credentials.get("apikey"),
    project_id=project_id,
    params=parameters
)

Set up a PromptTemplate to ask multiple questions. The retriever (the vector database) provides the "context" with the relevant documents, and the user query provides the "question."

template = """Generate a summary of the context that answers the question. Explain the answer in multiple steps if possible.
Answer style should match the context. Ideal Answer Length 2-3 sentences.\n\n{context}\nQuestion: {question}\nAnswer:
"""
prompt = ChatPromptTemplate.from_template(template)

Set up a helper function to format the docs:

def format_docs(docs):
    return "\n\n".join([d.page_content for d in docs])

Now set up a chain with the context, prompt, and LLM. The generative model processes the augmented context along with the user's question to produce a response.

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

Now you can ask multiple questions:

chain.invoke("Where is Think 2024?")

Output:

    '\nAnswer: The location of Think 2024 is Boston, Massachusetts, USA.'

Ask about IBM Concert next.

chain.invoke("What is IBM Concert?")

Output:

    'A) A new product announcement from IBM regarding their automation portfolio\n'

Finally, ask what IBM Think 2024 is.

chain.invoke("What is IBM Think 2024?")

Output:

    "A: IBM's annual user conference that brings together senior business and technology leaders from across industries for exploration, discussion, and innovation around artificial intelligence, data and automation capabilities, and hybrid cloud solutions."

You can continue asking more questions.

Summary and next steps

In this tutorial, you created a LangChain RAG system in Python with watsonx. You fetched 27 articles from https://www.ibm.com to create a vector store as context for an LLM to answer questions about the Think 2024 conference.

You can extend this pattern to build chatbots that answer questions using a custom knowledge base. See the LangChain documentation for more information and tutorials on RAG.

Try watsonx for free

Build an AI strategy for your business on one collaborative AI and data platform called IBM watsonx, which brings together new generative AI capabilities, powered by foundation models, and traditional machine learning into a powerful platform spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data.

Try watsonx.ai, the next-generation studio for AI builders.

Next steps

Explore more articles and tutorials about watsonx on IBM Developer.