Enhance LLM performance: Document chunking with watsonx

Retrieval augmented generation (RAG) is a cutting-edge technique in the field of language models that enhances the accuracy and efficiency of information retrieval by combining the strengths of retrieval-based and generation-based models. RAG-based language models can significantly improve the relevance and coherence of generated content by integrating external knowledge into the generation process. When building these models, it's crucial to format the input data in a way that allows for fast retrieval while also maintaining high accuracy. This brings us to the challenge of chunking, as the process of data ingestion to create large language models (LLM) often requires large documents that must be broken into smaller, meaningful parts.

Traditional methods for document chunking, such as relying on HTML document structures or using semantic cosine similarity, have their limitations. Instead, a more innovative approach involves using existing LLMs to autonomously decide how to chunk documents. This method not only simplifies the chunking process but also potentially enhances the overall coherence and relevance of the generated chunks.

This article explains how watsonx-driven chunking can result in quicker response times and more accurate retrieval results.

Overall process

The proposed chunking process involves several steps.

Step 1. Extract text and split into sentences

You begin by extracting the entire text from each document and splitting it into individual sentences. This step ensures that the text is broken down into manageable units for processing.

Figure 1. Separate by document, then break down into single sentences

Step 2. Initial chunk creation

Automatically create an initial chunk from the first sentence of the document. This serves as a starting point for the chunking process. Subsequent sentences are processed individually by the LLM, as explained in Step 4.

Figure 2. Creating the first chunk from the first sentence

Step 3. Generate metadata for chunks using watsonx

By using watsonx, you can ask the model to generate an appropriate title, ID, and summary for each created chunk that you pass in. This metadata helps in identifying and retrieving chunks efficiently.

Figure 3. System I/O diagram of chunk before and after passing into LLM

Figure 4. What a chunk actually looks like in the metadata, saved as a dictionary

Step 4. Contextual sentence processing

With the generated metadata as context that can be passed into watsonx, you process the remaining sentences one by one.

• If the LLM determines that a sentence fits the main idea of an existing chunk, the sentence is added to that chunk. You then must pass this chunk back into watsonx to create an updated summary and title to reflect the new chunk, essentially repeating Step 3.

• If the sentence does not fit any existing chunk, a new chunk is created using the process from Step 2.

Figure 5. System design flow of remaining sentences

Step 5. Repeat until completion

Continue this process until all sentences in the document have been processed and appropriately chunked.

At the end of this process, you should have multiple chunks, where each chunk is a group of sentences that each have semantically similar sentences decided by the LLM. Then, you can continue to perform exploratory analysis, data visualizations, and data cleansing before passing the data to the RAG-based model.

Advantages

There are several advantages to this approach:

• Context preservation: By allowing the LLM to decide the chunking based on context, this method ensures that the semantic meaning between chunk splits is maintained. This preservation of context enhances the coherence and relevance of the information retrieved.

• Future-proofing: As LLM technology continues to advance, the speed and accuracy of this chunking process will improve. This makes the method not only effective now but also increasingly beneficial in the future.

• Enhanced metadata: Generating titles, IDs, and summaries for each chunk provides a structured way to understand and retrieve specific information. This enhanced metadata aids in quicker identification and access to relevant chunks.

• Contextual accuracy: Using metadata as context helps the LLM make more informed decisions about which sentences belong to which chunks. This improves the accuracy and coherence of the chunking, as each chunk is more likely to be semantically consistent.

• Focused processing: The LLM can focus on smaller, more manageable sections of text, potentially enhancing its performance and accuracy. By dealing with chunks rather than entire documents, the LLM can provide more precise and relevant responses.

• Easier updates: Smaller chunks can be individually updated or modified without affecting the entire document. This modular approach to document management simplifies the process of making changes and updates.

• Parallel processing: Chunks can be processed in parallel, potentially speeding up the overall processing time. This advantage is particularly significant for large-scale document processing tasks.

Disadvantages

There are also several disadvantages to this process:

• Current speed: The process can be slow due to the need to wait for LLMs to generate responses for each sentence. However, this is a one-time effort for training, and the speed is expected to improve with advancements in LLM technology.

• Resource consumption: This method might require more computational resources due to the repeated calls to watsonx and the LLM. The increased resource consumption could be a limiting factor, especially for large-scale implementations.

• Dependency on LLM performance: The effectiveness of this chunking approach heavily depends on the performance and accuracy of the LLMs used. Any limitations or inaccuracies in the LLMs directly impact the quality of the chunking process.

Results from the project

By using watsonx for chunking, document retrieval times significantly decreased, thereby enhancing overall model performance. The following image shows that for all recall values (indicating the number of documents targeted for retrieval), there was an improvement with this chunking method.

Figure 6. Document retrieval performance

Additionally, the RAG LLM's model responses improved significantly after chunking, as illustrated in Figure 7 (using regular structure chunking) and Figure 8 (using watsonx for chunking). The response time for this particular question was over 70% faster, and the answers were much more concise and direct. This chunking method eliminates the repetition and wordiness seen with structure chunking, showcasing the effectiveness of LLMs in semantic chunking.

Figure 7. Response using HTML structured chunking

Figure 8. Response using watsonx chunking

Conclusion

Using LLMs for document chunking is an innovative approach that offers significant benefits in terms of context preservation, future-proofing, and improved metadata. While there are challenges related to speed, resource consumption, and dependency on LLM performance, these are expected to diminish as LLM technology advances.

This article explained how watsonx-driven chunking resulted in quicker response times and more accurate retrieval results, demonstrating the potential of this method for future applications.

To read more, take a look at this use case that discusses retrieval augmented generation with large language models from watsonx.ai.