IBM Developer

Article

How the ColBERT re-ranker model in a RAG system works

Understand late interaction and MaxSim scoring for smarter RAG pipelines

By SoumyaRanjan Swain

On this page

ColBERT (Contextualized Late Interaction over BERT) is a retrieval model that balances the speed of traditional methods like BM25 (Best Matching 25) with the semantic accuracy of BERT (Bidirectional Encoder Representations from Transformers), an open source deep learning model for natural language understanding. ColBERT uses late interaction and MaxSim scoring to rank documents by relevance, making it especially effective in retrieval-augmented generation (RAG) pipelines.

A re-ranker reorders a list of retrieved documents based on their relevance to a query. In a typical retrieval pipeline, 2 stages work together:

  1. Initial retrieval: A lightweight retriever (for example, BM25 or a sparse/lexical retriever) fetches a large set of candidate documents quickly.
  2. Re-ranking: A more sophisticated model reorders the retrieved candidates to improve relevance and accuracy.

Types of re-rankers

The following table describes different types of re-rankers and their features.

Type Strengths Weaknesses Example use cases
Traditional Fast, interpretable, lightweight Lack semantic understanding Basic search engines, initial filtering
Cross-encoders High accuracy, deep interaction Computationally expensive Document ranking for question answering (QA), passage retrieval
Bi-encoders Efficient, scalable Less accurate for fine-grained queries Large-scale retrieval, first-pass ranking
Late Interaction Models Fine-grained, efficient Moderate computational cost RAG systems, conversational artificial intelligence (AI)
Hybrid Best of both worlds Integration complexity Enterprise search, hybrid RAG systems

ColBERT re-ranker employs late interaction for scoring, which allows for efficient yet effective ranking of documents.

How ColBERT works

Unlike standard transformer-based retrievers, which calculate relevance scores by concatenating a query and a document into a single sequence, ColBERT uses late interaction. This means:

  • The query and document embeddings are computed independently.
  • The interaction happens later, during scoring, rather than during encoding.

This approach allows pre-computation of document embeddings, making retrieval much faster without significant loss in accuracy.

Architecture and workflow for ColBERT

The following image explains the working fundamentals of ColBERT re-ranker.

Architecture diagram: ColBERT independently encodes query and document tokens, then computes late interaction scores via MaxSim

ColBERT scoring example

The following example demonstrates how ColBERT works.

Assume a query and document as follows:

  • Query → Q: “capital city”
  • Document-1 →D1: “Paris is the capital and largest city”

Step 1. Query and document token embeddings

Each token in the query and document is transformed into a 3-dimensional embedding. For ease of calculation, assume each token maps to a 3-dimensional vector (in reality, embeddings are 512 to 1024 dimensional).

  • Query “Q”: “capital city” > Tokenization > ‘capital’, ‘city’
  • Document "D": "Paris is the capital and largest city" > Tokenization > 'Paris', 'is', 'the', 'capital', 'and', 'largest', 'city'

Step 2. Similarity calculation

For each query token, its similarity with all document tokens is calculated. ColBERT typically uses dot product as the similarity metric: sim(a,b)= a1*​b1​+a2*​b2​+a3*​b3​.

The following figure shows the similarity calculation of the ‘capital’ token.

Matrix table showing dot product similarity scores between the query token "capital" and each document token

The following figure shows a similar calculation for query token 2 “city”.

Matrix table showing dot product similarity scores between the query token "city" and each document token

Step 3. Sum of MaxSim

The overall score for the document is the sum of the MaxSim values for all query tokens.

Score of document-1(D1): Score(Q,D1)=MaxSim(“capital”)+MaxSim(“city”)=5+3=8

Assume there are three more documents D2, D3, D4. ColBERT re-ranker calculates the MaxSim score as follows:

  1. Score(Q,D2)=9
  2. Score(Q,D3)=5
  3. Score(Q,D4)=2

After reranking, the order is D2, D1, D3, D4.

Summary

ColBERT uses the late interaction and MaxSim scoring method to rank the documents based on their relevance to the query. This scoring method provides better retrieval because of:

  • Fine-grained matching: By computing token-to-token similarities and taking the maximum for each query token, ColBERT focuses on the most relevant parts of the document.
  • Context awareness: The embeddings capture the contextual meaning of words (for example, “capital” in different contexts).

These strengths make ColBERT especially effective in retrieval-augmented generation (RAG) pipelines, where precise and contextually rich document retrieval directly impacts the quality of generated answers.