How the ColBERT re-ranker model in a RAG system works

A re-ranker is a model or system that is used in information retrieval to reorder or refine a list of retrieved documents or items based on their relevance to a given query.

In a typical retrieval pipeline, the process consists of two stages:

1. Initial retrieval: A lightweight retriever (for example, BM25, or Best Matching 25, is a dense retriever) that fetches a large set of candidate documents quickly.
2. Re-ranking: A more sophisticated and computationally expensive model that reorders the retrieved candidates to improve relevance and accuracy.

ColBERT (Contextualized Late Interaction over BERT) is a retrieval model that is designed to strike a balance between the efficiency of traditional methods like BM25 and the accuracy of deep learning models like BERT, an open source deep learning model used for natural language understanding.

The ColBERT re-ranker is especially effective in retrieval-augmented generation (RAG) pipelines, where precise and contextually rich document retrieval directly impacts the quality of generated answers.

Types of re-rankers

Here are different types of re-rankers and their features.

ColBERT re-ranker employs late interaction for scoring, which allows for efficient yet effective ranking of documents.

How ColBERT works

Unlike standard transformer-based retrievers, which calculate relevance scores by concatenating a query and a document into a single sequence, ColBERT uses late interaction. This means:

• The query and document embeddings are computed independently.
• The interaction happens later, during scoring, rather than during encoding.

This approach allows pre-computation of document embeddings, making retrieval much faster without significant loss in accuracy.

Architecture and workflow for ColBERT

The following image explains the working fundamentals of ColBERT re-ranker.

ColBERT example

Let’s understand how it works through a toy example.

Let’s assume we have query and document as follows:

• Query → Q: “capital city”
• Document-1 →D1: “Paris is the capital and largest city”

Step 1. Query and document token embeddings

Each token in the query and document is transformed into a 3-dimensional embedding. (For ease of calculation, let’s assume each token is converted to 3-dimensional vector embeddings (in reality they would be 512 to 1024 dimensional).)

• Query “Q”: “capital city” > Tokenization > ‘capital’, ‘city’
• Document “D” : "Paris is the capital and largest city" > Tokenization > ‘Paris’, ‘is’, ‘the’, ‘capital’, ‘and’, ‘largest’, ‘city’

Step 2. Similarity calculation

For each query token, its similarity with all document tokens is calculated. ColBERT typically uses dot product as the similarity metric: sim(a,b)= a1*​b1​+a2*​b2​+a3*​b3​.

The following figure shows the similarity calculation of the ‘capital’ token.

The following figure shows a similar calculation for query token 2 “city”.

Step 3. Sum of MaxSim

The overall score for the document is the sum of the MaxSim values for all query tokens.

Score of document-1(D1): Score(Q,D1)=MaxSim(“capital”)+MaxSim(“city”)=5+3=8

Let’s say there are three more documents D2,D3,D4. ColBERT re-ranker calculates the MaxSim score as follows:

1. Score(Q,D2)=9
2. Score(Q,D3)=5
3. Score(Q,D4)=2

So after reranking the order will be D2,D1,D3,D4.

Conclusion

ColBERT uses the “late interaction” and “MaxSim” scoring method to rank the documents based on their relevance to the query. This scoring method furnishes better retrieval because of:

• Fine-grained matching: By computing token-to-token similarities and taking the maximum for each query token, ColBERT focuses on the most relevant parts of the document.
• Context awareness: The embeddings capture the contextual meaning of words (for example, “capital” in different contexts).

These strengths make ColBERT especially effective in retrieval-augmented generation (RAG) pipelines, where precise and contextually rich document retrieval directly impacts the quality of generated answers.