IBM Developer

Tutorial

Build a Langchain RAG application for PDF documents using Llama 3.1-405b in watsonx.ai

Retrieve documents to create a vector store as context for an LLM to answer questions

By Alex Seymour

On this page

In this tutorial, we’ll build a Retrieval Augmented Generation (RAG) application to answer questions on InstructLab using the meta-llama/llama-3-405b-instruct model available in watsonx.ai.

What is RAG? Retrieval Augmented Generation (RAG) is an AI technique that retrieves information from an external knowledge base to ground large language models (LLMs) on accurate, up-to-date information. Learn more about the RAG architectural pattern and other generative AI architectural patterns from the IBM Architecture Center.

Two fundamental technologies for working with large language models (LLMs) are Langchain and Jupyter notebooks.

  • Langchain is an open-source framework that provides developers with the building blocks necessary to work with large language models (LLMs).
  • Jupyter notebooks are great for learning how to build with LLMs as they provide a flexible and versatile evironment to experiment, prototype, and debug when issues occur.

Prerequisites

You need an IBM Cloud Account. Sign up for a free account here.

Steps

Step 1. Set up your environment

  1. Create and record an IBM Cloud API key

  2. Log into watsonx.ai using your IBM Cloud account.

  3. Create a watsonx.ai project.

  4. Create and connect a Watson Machine Learning (WML) service instance

  5. Create a Jupyter notebook on watsonx.ai or locally on your machine

Step 2. Install and import relevant libaries

We'll need a few libraries and modules for this tutorial.

pip install ibm_watsonx_ai
pip install ibm_watson_machine_learning
pip install langchain_ibm
pip install langchain_community
pip install langchain_text_splitters
pip install langchain_core
pip install chromadb
pip install pypdf

After installing the libraries, you can import these modules.

import getpass
import os

from ibm_watsonx_ai.metanames import EmbedTextParamsMetaNames
from ibm_watsonx_ai.foundation_models.utils.enums import EmbeddingTypes

from ibm_watson_machine_learning.foundation_models.utils.enums import ModelTypes
from ibm_watson_machine_learning.metanames import GenTextParamsMetaNames as GenParams

from langchain_ibm import WatsonxEmbeddings, WatsonxLLM
from langchain_community.document_loaders import PyPDFLoader

from langchain_text_splitters import RecursiveCharacterTextSplitter
from langchain.vectorstores import Chroma

from langchain_core.prompts import ChatPromptTemplate
from langchain_core.output_parsers import StrOutputParser
from langchain_core.runnables import RunnablePassthrough

Step 3. Provide the credentials

You must provide the credentials that are required to connect to the watsonx.ai platform.

Depending on the region of your provisioned service instance, use one of the following as your watsonx URL:

  • Dallas: https://us-south.ml.cloud.ibm.com
  • London: https://eu-gb.ml.cloud.ibm.com
  • Frankfurt: https://eu-de.ml.cloud.ibm.com
  • Tokyo: https://jp-tok.ml.cloud.ibm.com

To get your project ID, from your project, click the Manage tab. Then, copy the project ID from the Details section of the General page.

To get your API key, refer to the API keys section of IBM Cloud console.

watsonx_url = getpass.getpass("Please enter your watsonx URL (hit enter): ")
watsonx_project_id = getpass.getpass("Please enter your watsonx project ID (hit enter): ")
watsonx_api_key = getpass.getpass("Please enter your watsonx API ket (hit enter): ")

Step 4. Load the documents

The first step in a RAG pipeline is to load the documents into the application. In this example, we are loading a PDF that contains information related to Instruct Lab and watsonx.ai.

loader = PyPDFLoader("./documents/InstructLab and watsonx.ai FAQs.PDF")
documents = loader.load()

Step 5. Split the documents

The second step in a RAG application pipeline is to split the documents, which improves retrieval and the relevancy of data that is provided to the model.

text_splitter = RecursiveCharacterTextSplitter(
    chunk_size=300,
    chunk_overlap=100,
    length_function=len,
    is_separator_regex=False,
)
split_documents = text_splitter.split_documents(documents)

Step 6. Store the documents

The third step in a RAG pipeline is to vectorize the documents and store the embeddings within a vector store. In this example, we use IBMs slate-125m-english-rtrvr model hosted in watsonx.ai to generate the embeddings.

embed_params = {
    EmbedTextParamsMetaNames.TRUNCATE_INPUT_TOKENS: 3,
    EmbedTextParamsMetaNames.RETURN_OPTIONS: {"input_text": True},
}

embeddings = WatsonxEmbeddings(
    model_id="ibm/slate-125m-english-rtrvr",
    url=watsonx_url,
    project_id=watsonx_project_id,
    apikey=watsonx_api_key,
    params=embed_params,
)

vectorstore = Chroma.from_documents(documents=split_documents, embedding=embeddings)

Step 7. Retrieve the documents

The fourth step in a RAG pipeline is to retrieve the documents that are relevant to the users questions. To implement this, we will use the Langchain retriever interface.

retriever = vectorstore.as_retriever()

Step 8. Generate a response

We'll now combine all the elements to create a sequence that performs the following steps:

  1. Accepts a question
  2. Finds relevant documents
  3. Builds a prompt
  4. Sends the prompt to a model
  5. Interprets the model's response

For this process, we'll use the meta-llama/llama-3-405b-instruct model. However, any model that is compatible with watsonx.ai could be used as an alternative.

parameters = {
    GenParams.DECODING_METHOD: 'greedy',
    GenParams.TEMPERATURE: 1,
    GenParams.TOP_P: 1,
    GenParams.TOP_K: 1,
    GenParams.MIN_NEW_TOKENS: 10,
    GenParams.MAX_NEW_TOKENS: 2000,
    GenParams.REPETITION_PENALTY:1,
    GenParams.STOP_SEQUENCES:[],
    GenParams.RETURN_OPTIONS: {'input_tokens': True,'generated_tokens': True, 'token_logprobs': True, 'token_ranks': True, }
}

llm = WatsonxLLM(
    model_id="meta-llama/llama-3-405b-instruct",
    url=watsonx_url,
    apikey=watsonx_api_key,
    project_id=watsonx_project_id,
    params=parameters
)

template = """<|begin_of_text|><|start_header_id|>system<|end_header_id|>

Given the document and the current conversation between a user and an assistant, your task is as follows: answer any user query by using information from the document. Always answer as helpfully as possible, while being safe. When the question cannot be answered using the context or document, output the following response: "I cannot answer that question based on the provided document.".

Your answers should not include any harmful, unethical, racist, sexist, toxic, dangerous, or illegal content. Please ensure that your responses are socially unbiased and positive in nature.

If a question does not make any sense, or is not factually coherent, explain why instead of answering something not correct. If you don't know the answer to a question, please don't share false information.<|eot_id|><|start_header_id|>user<|end_header_id|>

{context}<|eot_id|><|start_header_id|>user<|end_header_id|>

{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>

"""

prompt = ChatPromptTemplate.from_template(template)

def format_docs(docs):
    return "\n\n".join([d.page_content for d in docs])

chain = (
    {"context": retriever | format_docs, "question": RunnablePassthrough()}
    | prompt
    | llm
    | StrOutputParser()
)

chain.invoke("What is Instruct Lab?")

Ouput:

According to the provided document, InstructLab is a way to facilitate Large Language model development through collaboration among the open-source community. It consists of a Command Line Interface (CLI) and a backend for synthetic data generation and model training.

Summary and next steps

In this tutorial, we built a RAG application to answer questions about InstructLab using the meta-llama/llama-3-405b-instruct model now available in watsonx.ai.

If you want to learn how to use the watsonx Prompt Lab to build a RAG application in a no-code manner to answer questions about IBM securities, see this tutorial. Or, if you want to learn how to build a LangChain RAG system for web data using Python, see this tutorial.

In this tutorial, we used the SaaS offering of Llama models in watsonx.ai. Additionally, Llama 2 and 3 are available for multi-cloud deployments (on AWS, Azure, or GCP) and also on-premises. A self-deployed release of the Llama 3.1 series is also coming soon that will allow developers to deploy these models on the platform of your choice (multi-cloud or on-premises, with no vendor lock-in).

Try watsonx for free

Build an AI strategy for your business on one collaborative AI and data platform called IBM watsonx, which brings together new generative AI capabilities, powered by foundation models, and traditional machine learning into a powerful platform spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data. These models are accessible to all as many no-code and low-code options are available for beginners.

Try watsonx.ai, the next-generation studio for AI builders.

Next steps

Explore more articles and tutorials about watsonx on IBM Developer.