Use the no-code RAG solution in watsonx Prompt Lab to build a RAG app with Llama3.1-405b

Most enterprises that are using generative AI today are augmenting LLMs with knowledge search and documents. Content-grounded Q&A is one of the most popular generative AI use cases for enterprise and a game-changer for reducing hallucination and providing more accurate responses. Whether this is a customer care use case, a call center, IT automation, or operational excellence.

IBM watsonx recently released “Chat with Documents”, a simple no-code Retrieval Augmented Generation (RAG) solution that lets you upload a collection of documents (PDF, PPT, text, or doc) or connect your LLM to a set of thousands of documents coded in a vector database.

In this tutorial, we’ll build a RAG application to augment meta-llama/llama-3-405b-instruct model with a sample input PDF. The PDF we are using here is the 2023 IBM-10K to ground the responds from the llama model.

When do you use RAG?

What is RAG? Retrieval Augmented Generation (RAG) is an AI technique that retrieves information from an external knowledge base to ground large language models (LLMs) on accurate, up-to-date information. Here are some reasons you might want to use RAG:

• Access to up-to-date information. The knowledge of LLMs is limited to what they were exposed to during pre-training. With RAG, you can ground the LLM to the latest data feeds, making it perfect for real-time use cases.
• Incorporating proprietary data. LLMs weren't exposed to your proprietary enterprise data (data about your users or your specific domain) during their training and have no knowledge of your company data. With RAG, you can expose the LLM to company data that matters.
• Minimizing hallucinations. LLMs are not accurate knowledge sources and often respond with made-up answers. With RAG, you can minimize hallucinations by grounding the model to your data.
• Rapid comparison of LLMs. RAG applications allow you to rapidly compare different LLMs for your target use case and on your data, without the need to first train them on data (avoiding the upfront cost and complexity of pre-training or fine-tuning).
• Control over the knowledge the LLM is exposed to. RAG applications let you add or remove data without changing the model. Company policies change, customers' data changes, and unlearning a piece of data from a pre-trained model is expensive. With RAG, it's much easier to remove data points from the knowledge your LLM is exposed to.

Learn more about the RAG architectural pattern and other generative AI architectural patterns from the IBM Architecture Center.

Prerequisites

• You need an IBM Cloud Account. Sign up for a free account here.
• You also need a watsonx.ai account. Sign up for a free account here.

Steps

Step 1. Open Prompt Lab and chat with the Llama 3.1 405B model

1. Log in to watsonx.ai.
2. Create a watsonx.ai project.
3. Select Open Prompt Lab from the main page.

You can chat with Llama 3.1 405B right from Prompt Lab to test it out.

Step 2. Load your grounding documents

In this example, we are loading a PDF containing information related to 2023 IBM SEC Form 10-K Annual Report.

To do this, select Grounding data from the Prompt lab, and follow the wizard to upload your PDF. Point at a COS or S3 bucket, and Prompt Lab processes the applicable files that are found there, automatically parses and converts them to text, chunks them, uses the optimized embedding models, and finally indexes those embeddings -- all in one click.

Now your Llama 3.1-405B model is ready to augment its answers with the knowledge provided in the PDF. This approach helps minimize "hallucinations," the made-up answers that LLMs are prone to.

This was an example of one single PDF, but what if you have thousands of docs? Instead of keeping them all in memory, you might want to save them in a vector database for lower latency and reusability. IBM watsonx has just released a feature that makes it easy to connect your LLM to a vector database by using a variety of connectors, such as Milvus or Elasticsearch, instead of uploading a collection of static documents.

Step 3. Save your prompt

Once you are done, you can save your prompt as a session, template, or a notebook and share with others to collaborate on or use it.

Summary and next steps

In this tutorial, we used the watsonx Prompt Lab to build a RAG application in a no-code manner to answer questions about IBM securities using the meta-llama/llama-3-405b-instruct model, which is now available in watsonx.ai.

If you want to learn how to use Langchain and Python code to build this RAG app, see this tutorial. Or, if you want to learn how to build a LangChain RAG system for web data using Python, see this tutorial.

In this tutorial, we used the SaaS offering of Llama models in watsonx.ai. Additionally, Llama 2 and 3 are available for multi-cloud deployments (on AWS, Azure, or GCP) and also on-premises. A self-deployed release of the Llama 3.1 series is also coming soon that will allow developers to deploy these models on the platform of your choice (multi-cloud or on-premises, with no vendor lock-in).

Try watsonx for free

Build an AI strategy for your business on one collaborative AI and data platform called IBM watsonx, which brings together new generative AI capabilities, powered by foundation models, and traditional machine learning into a powerful platform spanning the AI lifecycle. With watsonx.ai, you can train, validate, tune and deploy models with ease and build AI applications in a fraction of the time with a fraction of the data. These models are accessible to all as many no-code and low-code options are available for beginners.

Try watsonx.ai, the next-generation studio for AI builders.

Next steps

Explore more articles and tutorials about watsonx on IBM Developer.