Article
Improve enterprise search relevance with LLM-augmented retrieval
A practical guide to improving enterprise search with LLMs, semantic structuring, intent-aware retrieval, ELSER, vector search, reranking, RAG, and benchmark-driven optimizationLarge-scale enterprise search systems often return irrelevant results, not because the content is missing, but because the search pipeline does not understand what the user needs. A structured retrieval framework that uses large language models (LLMs) can improve search at three stages: Query, Search, and Results.
Consider a compliance analyst who searches an enterprise knowledge base for "what happens if a vendor misses the service level agreement (SLA)?" The relevant document might use the phrase "consequences of service level breaches." Because the query and document use different terms, a keyword-based search might not return the document. The search system needs semantic understanding to recognize that the phrases have similar meanings.
A structured retrieval improvement framework addresses this problem across three stages:
- Query: Transform and enrich the user's query before sending it to the search engine.
- Search: Improve how the search system indexes documents and matches documents to the query.
- Results: Reorder, filter, and synthesize the retrieved results to produce a precise answer.
This article explains each stage of the framework, when to apply each improvement, and how to run a structured pilot. The pilot uses benchmark results to select and evaluate retrieval improvements instead of relying on assumptions.
Enterprise use cases for LLM-augmented retrieval
LLM-augmented retrieval is useful when users and documents use different terms or when the information needed to answer a question is spread across multiple sources.
Common enterprise use cases include:
- Customer support portals: Find relevant policy and procedure documents for customer questions written in natural language.
- Legal and compliance search: Match questions about obligations and risks to relevant clauses in contracts and regulations.
- Internal knowledge bases: Help employees find HR policies, IT procedures, and project documentation without knowing the exact terminology.
- Financial document analysis: Find specific figures, disclosures, and projections in lengthy financial reports.
- Technical documentation: Match developer questions to the appropriate API reference, guide, or code example across large documentation collections.
In this article, you apply the framework to improve enterprise search relevance. The same Query, Search, and Results stages can support each use case. Only the document collection and benchmark queries need to change.
When to use RAG and retrieval levers
Retrieval-augmented generation (RAG) works well when a clear user query can retrieve the relevant documents and an LLM can use those documents to generate an answer. RAG is useful for focused questions that search structured knowledge bases.
LLM retrieval levers are useful when the user query is unclear, lacks context, uses different terminology, or contains problematic input. Retrieval levers can also improve how the search system indexes, retrieves, and ranks documents.
Many enterprise search systems use both approaches. Retrieval levers improve the documents that the search system retrieves, while RAG uses the retrieved documents to generate an answer. Together, the two approaches can improve the relevance and accuracy of enterprise search results.
Prerequisites
Before you begin, make sure that you have the following resources:
- An enterprise search system, such as IBM Watson Discovery, Elasticsearch, or OpenSearch.
- A benchmark set with test queries and expected results. You use the benchmark set to measure retrieval quality before and after changes.
- Access to an LLM API, such as IBM watsonx.ai, to implement query transformation and answer generation.
- Information retrieval metrics, such as normalized discounted cumulative gain (nDCG), mean reciprocal rank (MRR), or Precision@k, to evaluate retrieval performance.
Architecture of the retrieval improvement pipeline
The retrieval improvement framework organizes search improvements into five stages. You can improve each stage independently and use the benchmark set to measure the effect of each change.
Content → Semantic Structuring → Query → Search → Results
The Content stage runs before the search system processes user queries. In the Semantic Structuring stage, the system assigns a functional type to each content chunk and maps each user query to a specific intent. The search system uses the chunk type and query intent to determine which content chunks are relevant before running the search.
The Query, Search, and Results stages run when the user submits a query. LLM-based retrieval levers can improve each stage. The Query and Semantic Structuring stages often provide the most opportunities to improve retrieval quality.
The following diagram shows how the retrieval stages and query-time levers work together across different content collections.
Pipeline stages
This article covers each stage of the retrieval improvement pipeline. You can implement one stage at a time and use the benchmark set to measure the effect of each change before moving to the next stage.
Stage 0 runs offline and prepares the content for retrieval. Stages 1, 2, and 3 run when a user submits a query. These stages improve the query, search the appropriate content collections, and refine the search results.
| Stage | Name | When it runs | Primary focus |
|---|---|---|---|
| Stage 0 | Content standardization and semantic structuring | Offline | Clean and structure indexed content, assign chunk types, and map user intents |
| Stage 1 | Query understanding | Query time | Transform and enrich the user query |
| Stage 2 | Search | Query time | Search the appropriate content collections |
| Stage 3 | Results | Query time | Rerank and filter the search results, then generate the final answer |
Stage 0: Content standardization and semantic structuring
Before a user submits a query, the search system needs clean and well-structured indexed content. Stage 0 runs offline and focuses on content standardization and semantic structuring.
Content standardization prepares documents for indexing. The process converts formats such as HTML, PDF, and DOCX into consistent text, standardizes terminology across sources, and removes unnecessary content. Poorly prepared or incorrectly indexed content can reduce the quality of later retrieval stages.
Semantic structuring organizes content based on its purpose. The system assigns each content chunk a chunk type, such as Concept, Diagnostic, Playbook, or Artifact. The system also classifies each user query by intent, such as learn, assess, or create.
At query time, the search system uses the query intent to identify the appropriate chunk types. This approach helps the search system retrieve content based on the type of information the user needs, not only on semantic similarity.
Note: Complete this stage before you improve any query-time retrieval levers. Content standardization and semantic structuring provide the foundation for all later retrieval improvements.
Stage 1: Query understanding
The Query stage focuses on understanding and improving the user query before the search system processes it. If the search system misunderstands the user query, later retrieval improvements might not find the relevant content.
Summarize long queries: Enterprise users might enter long paragraphs or email threads into a search box. Long queries can make it difficult for keyword and semantic search systems to identify the main intent. An LLM can summarize a long query into a concise, search-optimized query while preserving the user's intent.
Example input: "We need to know about the process that the team should follow when a critical production system goes down in the middle of the night and we can't reach the primary on-call engineer."
Summarized query: "critical incident escalation procedure when primary on-call is unavailable"
Rephrase the query: Query rephrasing converts unclear or informal language into terms that are more likely to match the indexed content.
Example input: "how do I get my request pushed through faster"
Rephrased query: "expedited approval process for service requests"
Generate an answer, then search with HyDE: Hypothetical Document Embeddings (HyDE) uses an LLM to generate a hypothetical answer to the user query. The search system creates an embedding from the hypothetical answer and uses the embedding to find relevant documents. HyDE can reduce the difference between short user queries and longer, detailed documents. HyDE is especially useful when users ask short questions against a collection of long documents.
Extract keywords and entities: Keyword and entity extraction converts a natural language query into structured search terms. The search system can identify information such as product names, people, dates, regulation codes, and domain-specific terms. The search system can use the extracted keywords and entities to filter results or give more weight to highly relevant documents.
Expand the query with related terms: Semantic query expansion adds related terms to a user query to improve retrieval. For example, a query for "vehicle emissions" might also retrieve documents about "carbon footprint," "exhaust standards," and "fuel efficiency regulations." The search system can use embedding similarity or an LLM to identify related terms. Semantic query expansion is useful when users and document authors use different terminology.
Check query topic relevance: Some user queries might be outside the scope of the search system. A topic relevance classifier identifies these queries and directs users to an appropriate response instead of returning irrelevant results.
Handle profanity: Enterprise search systems need to handle profanity or inappropriate language in user queries. The search system can identify this language and prevent inappropriate or sensitive content from appearing in the results.
Identify query intent: Intent classification identifies the type of information the user needs, such as a factual answer, procedure, comparison, navigation link, or summary. The search system can use the identified intent to select an appropriate retrieval strategy. For example, the search system can generate an answer for a factual question or return a ranked list of documents for a navigation query.
Stage 2: Search
After the search system receives a well-formed query, the Search stage identifies the documents that can provide the relevant information.
Use domain-specific dictionaries: Domain-specific dictionaries and ontologies help the search system recognize specialized terms, such as medical codes, legal terms, internal product names, and regulatory identifiers. The dictionaries help match user queries with relevant documents when users and document authors use different terminology.
Improve relevance with LLM training: LLM-enhanced relevance training improves how the search system ranks documents. The retrieval model can learn from query and document pairs that show which documents are relevant to specific queries. Training data can come from search usage logs, human relevance feedback, or synthetic query and document pairs generated by an LLM. This training helps the retrieval model better match the content collection and user queries.
Use vector search: Vector search uses semantic similarity to retrieve documents based on meaning instead of exact keyword matches. Vector search can find relevant documents when user queries use different wording, paraphrases, or languages. A hybrid search combines keyword search, such as BM25, with vector search. The search system combines the keyword and vector scores to improve both exact keyword matching and semantic relevance.
Stage 3: Results
After the search system retrieves a set of candidate documents, the Results stage determines which documents and answers the user sees.
Rerank the results: A cross-encoder reranker evaluates the retrieved documents again and assigns more accurate relevance scores. The initial retrieval step focuses on finding potentially relevant documents. The reranker focuses on placing the most relevant documents at the top of the results. Reranking can improve search precision without changing the search index or query processing. The reranker works with the results from the existing retrieval system.
Generate an answer: In a RAG system, the LLM uses the highest-ranked documents to generate a direct answer based on the retrieved content. The generated answer should include references to the source documents. Source references help users verify the answer and help auditors trace the information to its source.
Filter and rank by keywords and entities: The search system can use keywords and entities from the original user query to filter or rank the final results. Documents that contain the specific keywords or entities can rank higher than documents that are related to the topic but do not contain the requested information.
Pilot methodology
Use the following five-step pilot methodology to select, implement, and evaluate retrieval improvements.
Step 1. Set up the test environment
Use a development or staging environment, such as IBM Cloud DEV, that reflects the production search environment. Run all retrieval experiments in the test environment to avoid affecting production users.
Step 2. Analyze the benchmark set
Review the benchmark queries and expected results. Identify common retrieval problems by asking:
- Which types of queries fail most often?
- Does the search system fail to retrieve the correct document?
- Does the search system retrieve the correct document but rank another document higher?
- Do queries fail because of different terminology, unclear wording, or topics outside the scope of the search system?
Use the benchmark analysis to identify which retrieval levers can address the observed problems.
Step 3. Select retrieval levers
Select one or more LLM-based retrieval levers based on the benchmark results. Consider the expected improvement, implementation complexity, latency, and infrastructure requirements.
Start with retrieval levers that are practical to implement and directly address the identified retrieval problems. For example, query rephrasing and HyDE use an LLM to improve the query or search representation. Reranking uses a separate model to improve the order of retrieved documents without changing the search index.
Step 4. Implement the retrieval levers
Add the selected retrieval levers to the existing search system. Implement each retrieval lever as a separate pipeline component so that you can enable or disable each component during A/B testing.
Step 5. Evaluate retrieval performance
Evaluate the updated search system against the benchmark set by using standard information retrieval metrics.
| Metric | What it measures |
|---|---|
| nDCG@k | Quality of the ranked results within the top k positions |
| MRR | Position of the first relevant result |
| Precision@k | Proportion of relevant results within the top k positions |
| Recall@k | Proportion of all relevant documents retrieved within the top k positions |
Compare the results for each query type to identify where each retrieval lever improves performance and where the retrieval lever does not help. A retrieval lever might improve some query types while reducing the quality of others. The benchmark results help identify these differences.
Compare Golden Answer retrieval
For the combined collection approach, measure how many Golden Answers the search system retrieves in the top 3, top 5, and top 10 results.
Also compare:
- Included queries: Queries where the experiment retrieves a Golden Answer that the original search system did not retrieve.
- Missed queries: Queries where the original search system retrieved a Golden Answer but the experiment did not.
This comparison shows whether the combined collection approach improves retrieval and identifies queries where retrieval performance decreases.
Prioritization guide
If you are starting from scratch, implement the retrieval improvements in the following order:
- Standardize content: Clean and standardize the content before indexing. Poor content quality can reduce retrieval performance at every later stage.
- Assign chunk types: Assign a functional type to each content chunk. Chunk types provide the structure needed for intent-based retrieval.
- Implement intent mapping: Classify each user query by intent and map the intent to the appropriate chunk types before retrieval.
- Implement HyDE or query rephrasing: Transform each user query to improve the search representation and retrieve more relevant content.
- Add vector search and hybrid retrieval: Combine semantic vector search with keyword search to improve retrieval coverage.
- Add a cross-encoder reranker: Rerank the retrieved documents to place the most relevant documents higher in the results.
- Improve relevance with LLM training: Train the retrieval model with relevant query and document pairs. Usage logs can provide training data as the search system collects more user interactions.
Implement ELSER search, reranking, and intent-aware retrieval
The following Python code shows how to implement key components of the retrieval framework. The code covers three tasks:
- Search an Elasticsearch index by using the Elastic Learned Sparse Encoder (ELSER) model.
- Rerank the retrieved documents by using IBM watsonx.ai.
- Map user intent to chunk types to filter the content that the search system retrieves.
The implementation tasks follow:
Configure dependencies: Import the required libraries and define the configuration values.
import logging from elasticsearch import Elasticsearch from typing import Dict, List, Optional, Any from elasticsearch.exceptions import ApiError, TransportError from ibm_watsonx_ai.foundation_models import Rerank from ibm_watsonx_ai import Credentials, APIClient import re logger = logging.getLogger(__name__) # Keys stripped from metadata before returning results to the caller remove_keys = [ 'author', 'last_updated', 'source', 'token_count', 'version', 'word_count', 'chunk_id', 'source_document_id' ] # Connection parameters. Supply via environment variables or a secrets manager. # ibm_cloud_url = "https://<your-watsonx-region>.ml.cloud.ibm.com" # watson_ai_project_id = "<your-project-id>" # watson_ai_api_key = "<your-api-key>"Split text for reranking
Stage mapping: This utility is part of Stage 0: Content standardization and semantic structuring. The utility splits content into appropriately sized chunks before the reranker processes the content. Properly sized chunks help the cross-encoder reranker produce accurate scores in Stage 3: Results.
Cross-encoder rerankers have a limited token window. Splitting long content chunks into smaller passages helps the reranker score the content more accurately. After scoring, the highest passage score becomes the score for the original content chunk.
def split_into_pieces(text: str, max_chars: int = 3500) -> List"""Split text on sentence boundaries, staying under max_chars.""" sentences = re.split(r'(?<=[.!?])\s+', text.strip()) pieces = [] current = "" for sentence in sentences: if len(current) + len(sentence) + 1 <= max_chars: current = (current + " " + sentence).strip() else: if current: pieces.append(current) if len(sentence) > max_chars: for j in range(0, len(sentence), max_chars): pieces.append(sentence[j:j + max_chars]) else: current = sentence if current: pieces.append(current) return pieces if pieces else [text[:max_chars]]Rerank results with watsonx.ai
Stage mapping: This function implements the reranking step in Stage 3: Results.
The function uses the
cross-encoder/ms-marco-minilm-l-12-v2model to score the passages from the retrieved content chunks. The function keeps the highest passage score for each original content chunk, sorts the content chunks by score, and returns the top results.def rerank_results( query: str, contents: List[Dict[str, Any]], watson_ai_api_key: str, project_id: str, top_k: int = 10 ) -> List[Dict[str, Any]]: if not contents: return contents credentials = Credentials( url="https://<your-watsonx-region>.ml.cloud.ibm.com", api_key=watson_ai_api_key ) client = APIClient(credentials, project_id=project_id) reranker_model = Rerank( model_id="cross-encoder/ms-marco-minilm-l-12-v2", api_client=client, ) try: # Expand each chunk into sentence-boundary passages expanded = [] for i, item in enumerate(contents): pieces = split_into_pieces(item["chunk"], max_chars=1200) for piece in pieces: expanded.append((i, piece)) passages = [{"text": piece} for _, piece in expanded] response = reranker_model.generate( query=query, inputs=passages, params={ "truncate_input_tokens": 512, "return_options": {"top_n": len(passages)} } ) logger.info(f"Rerank response: {response}") # Promote the best passage score back to its parent chunk best_scores = {} for r in response["results"]: orig_idx = expanded[r["index"]][0] score = float(r["score"]) if orig_idx not in best_scores or score > best_scoresbest_scores[orig_idx] = score reranked = sorted( [{**contents[i], "score": score} for i, score in best_scores.items()], key=lambda x: x["score"], reverse=True ) return reranked[:top_k] except Exception as e: logger.error(f"Reranking error: {e}") return contentsSearch with ELSER and chunk type filtering
Stage mapping: This function implements Stage 2: Search. The
chunk_typesparameter uses the intent and chunk type mapping defined in Stage 0: Content standardization and semantic structuring.The function searches the Elasticsearch ELSER index by using
text_expansion. Whenchunk_typesis provided, the function filters the search results by the specified chunk types. The function then orders the results based on the chunk type sequence for the user intent. For example, Concept chunks can appear before Diagnostic chunks.def elser_search_custom( es_client: Elasticsearch, index_name: str, query: str, model_name: str, chunk_types: Optional[List[str]] = None, size: int = 20 ) -> Dict[str, Any]: """ Perform semantic search using ELSER with optional chunk-type filtering and canonical intent-order resorting. """ filter_clause = [] if chunk_types: filter_clause.append({ "terms": {"metadata.chunk_type.keyword": chunk_types} }) try: resp = es_client.search( index=index_name, size=size, query={ "bool": { "must": { "text_expansion": { "ml.tokens": { "model_text": query, "model_id": model_name } } }, "filter": filter_clause } }, _source=["text_field", "metadata"] ) chunks, metadatas = [], [] for hit in resp.get("hits", {}).get("hits", []): src = hit["_source"] chunks.append(src.get("text_field", "")) metadata = src.get("metadata", {}) for k in remove_keys: metadata.pop(k, None) metadatas.append(metadata) # Reorder results by the defined chunk-type sequence if chunk_types: order_index = {ct: i for i, ct in enumerate(chunk_types)} paired = list(zip(chunks, metadatas)) paired.sort( key=lambda x: order_index.get( x[1].get("chunk_type"), len(order_index) ) ) chunks, metadatas = map(list, zip(*paired)) if paired else ([], []) return { "content": [ {"metadata": m, "chunk": c} for c, m in zip(chunks, metadatas) ], "index_name": index_name, "status": "search completed" } except (ApiError, TransportError) as e: error_msg = f"Elasticsearch error: {str(e)}" logger.error(error_msg) except ValueError as e: error_msg = f"Validation error: {str(e)}" logger.error(error_msg) except Exception as e: error_msg = f"Unexpected error: {str(e)}" logger.error(error_msg) return { "content": [{"metadata": "", "chunk": ""}], "index_name": index_name, "status": error_msg }Define intent-to-chunk-type routing
Stage mapping: The
INTENT_RULESdictionary is part of Stage 0: Content standardization and semantic structuring. The Stage 2: Search function uses these rules at query time to filter content by chunk type.INTENT_RULESmaps each user intent to two groups of chunk types:- Primary: Chunk types that the search system retrieves first and uses for the main response.
Secondary: Chunk types that the search system retrieves for additional context when the primary results are not sufficient.
The
INTENT_RULESdictionary connects user intent classification to chunk type filtering.INTENT_RULES = { "learn": { "primary": ["Concept", "Framework", "Principle"], "secondary": ["Example", "Diagnostic", "Conversation_Guide", "Contextual_Adaptor", "Artifact_Template"] }, "assess": { "primary": ["Concept", "Framework", "Diagnostic", "Principle"], "secondary": ["Red_Flags", "Contextual_Adaptor"] }, "interpret": { "primary": ["Concept", "Framework", "Principle", "Red_Flags"], "secondary": ["Example", "Diagnostic", "Contextual_Adaptor"] }, "validate": { "primary": ["Concept", "Diagnostic", "Principle", "Framework", "Red_Flags"], "secondary": ["Playbook", "Ethical_Boundaries", "Contextual_Adaptor"] }, "decide": { "primary": ["Framework", "Principle", "Playbook", "Contextual_Adaptor"], "secondary": ["Concept", "Diagnostic", "Ethical_Boundaries"] }, "act": { "primary": ["Playbook", "Conversation_Guide", "Artifact_Template", "Principle"], "secondary": ["Concept", "Framework", "Contextual_Adaptor"] }, "reflect": { "primary": ["Concept", "Principle", "Example", "Calls_To_Action", "Red_Flags"], "secondary": ["Playbook", "Contextual_Adaptor"] } }
Resolve chunk types for multiple intents: A user query can have more than one intent. The
get_chunk_types_by_levelfunction combines the chunk types for all identified intents.The function removes duplicate chunk types while keeping the defined order. The function returns the combined chunk types for either the
primaryorsecondaryretrieval level.def get_chunk_types_by_level(intents: list[str], level: str) -> list[str]: """ Return deduplicated chunk types for a given level ('primary' or 'secondary') across all provided intents, preserving declaration order. """ seen = set() result = [] for intent in intents: rule = INTENT_RULES.get(intent.lower(), {}) for chunk_type in rule.get(level, []): if chunk_type not in seen: seen.add(chunk_type) result.append(chunk_type) return result
Query-time retrieval flow
At query time, the retrieval pipeline processes the user query through the following steps:
- The intent classifier identifies one or more user intents, such as
assessanddecide. - The
get_chunk_types_by_levelfunction identifies the primary and secondary chunk types for the user intents. - The
elser_search_customfunction searches the primary and secondary chunk types in parallel. - The retrieval pipeline merges the search results.
- If the merged results do not meet the required threshold, the
elser_search_customfunction searches the full index without chunk type filtering. - The
rerank_resultsfunction reranks the combined candidate chunks and returns the top results. - The LLM uses the top-ranked chunks to generate the final answer.
user query
│
▼
intent classifier → ["assess", "decide"]
│
▼
get_chunk_types_by_level(intents, "primary") + get_chunk_types_by_level(intents, "secondary")
│ │
▼ (parallel) ▼ (parallel)
elser_search_custom(…, chunk_types=primary) elser_search_custom(…, chunk_types=secondary)
│ │
└─────────────────── _merge_results ────────────────┘
│
▼ (fallback if merged count < threshold)
elser_search_custom(…, chunk_types=None) ← full-index scan
│
▼
rerank_results(query, merged_candidates, top_k=10)
│
▼
top-k reranked chunks → LLM answer synthesis
Use full-index search as a fallback
Stage mapping: This fallback is part of Stage 2: Search. The search system uses the fallback when intent-based filtering returns too few results.
Intent-based filtering can improve precision but might reduce recall for unusual queries or small content collections. When the primary and secondary searches return too few results, the search system runs an ELSER search across the full index without chunk type filtering.
The full-index search helps the search system find relevant content that the intent-based filters might have excluded.
Configure the fallback search
Set a threshold for the minimum number of results returned by the primary and secondary searches. The default threshold is three results.
If the combined results contain fewer than three results, the search system runs an ELSER search across the full index without a chunk_type filter. The search system then sends the fallback results to the reranker.
FALLBACK_THRESHOLD = 3 # Run full-index search if fewer results are returned
def search_with_fallback(
es_client: Elasticsearch,
index_name: str,
query: str,
model_name: str,
primary_types: List[str],
secondary_types: List[str],
size: int = 20,
) -> Dict[str, Any]:
"""
Run primary and secondary intent-filtered searches in parallel.
If the combined result count is below FALLBACK_THRESHOLD,
run an unfiltered full-index ELSER search.
"""
empty = {"content": [], "index_name": index_name}
with ThreadPoolExecutor(max_workers=2) as executor:
f_primary = executor.submit(
elser_search_custom,
es_client, index_name, query, model_name, primary_types, size,
) if primary_types else None
f_secondary = executor.submit(
elser_search_custom,
es_client, index_name, query, model_name, secondary_types, size,
) if secondary_types else None
primary_results = f_primary.result() if f_primary else empty
secondary_results = f_secondary.result() if f_secondary else empty
merged = _merge_results(primary_results, secondary_results)
if len(merged["content"]) < FALLBACK_THRESHOLD:
logger.warning(
f"Intent-filtered results below threshold ({len(merged['content'])} < "
f"{FALLBACK_THRESHOLD}). Falling back to full-index ELSER search."
)
fallback = elser_search_custom(
es_client,
index_name,
query,
model_name,
chunk_types=None,
size=size,
)
return {**fallback, "fallback_used": True}
return {**merged, "fallback_used": False}
The fallback_used flag tells the logging and evaluation systems whether the search system used the full-index fallback. A higher fallback rate can indicate that the intent classifier or the distribution of chunk types in the index needs review.
Deploy the retrieval tool to IBM watsonx Orchestrate
Package the retrieval function as a watsonx Orchestrate tool. Use the @tool decorator from ibm_watsonx_orchestrate to define the tool entry point.
After you register the tool in a watsonx Orchestrate environment, an agent can call the retrieval tool at runtime. Named connections provide the credentials that the retrieval tool needs.
Project structure
The project files have the following roles:
orchestrate_tool.py: Defines the@toolentry point.requirements.txt: Lists the required Python packages.utils/elasticsearch_search.py: Containselser_search_customandrerank_results.utils/rules.py: ContainsINTENT_RULESandget_chunk_types_by_level.utils/ssl_cert.py: Containsget_ca_bundle_path.
search_tool1/
├── orchestrate_tool.py
├── requirements.txt
└── utils/
├── elasticsearch_search.py
├── rules.py
└── ssl_cert.py
Create the watsonx Orchestrate retrieval tool
The following code defines the retrieval function as a watsonx Orchestrate tool. The tool retrieves credentials from configured connections, determines the chunk types for the user intent, searches Elasticsearch with ELSER, applies a full-index fallback when needed, and reranks the results with watsonx.ai.
The _merge_results function combines the primary and secondary search results and removes duplicate content before reranking.
from typing import List, Dict, Any
import logging
import os
from concurrent.futures import ThreadPoolExecutor
from elasticsearch import Elasticsearch
from ibm_watsonx_orchestrate.agent_builder.tools import tool
from ibm_watsonx_orchestrate.run import connections
from ibm_watsonx_orchestrate.client.connections import ConnectionType
from utils.rules import get_chunk_types_by_level
from utils.ssl_cert import get_ca_bundle_path
from utils.elasticsearch_search import elser_search_custom, rerank_results
logging.basicConfig(
level=logging.INFO,
format="%(asctime)s - %(name)s - %(levelname)s - %(message)s"
)
logger = logging.getLogger(__name__)
ES_APP_ID = "es_client"
WX_APP_ID = "Watson_api"
DEFAULT_INDEX = "<your-index-name>"
DEFAULT_MODEL = ".elser_model_2"
ES_HOST = "https://<your-es-host>:30791"
RERANK_TOP_K = 10
FALLBACK_THRESHOLD = 3
is_called_from_orchestrate = True
def _merge_results(primary: Dict, secondary: Dict) -> Dict:
"""Merge primary and secondary results and remove duplicates."""
seen, merged = set(), []
def _key(item):
meta = item.get("metadata", {})
return meta.get("id") or item.get("chunk", "")[:120]
for item in primary.get("content", []):
k = _key(item)
if k not in seen:
seen.add(k)
merged.append(item)
for item in secondary.get("content", []):
k = _key(item)
if k not in seen:
seen.add(k)
merged.append(item)
return {
"content": merged,
"index_name": primary.get("index_name"),
"status": "ok",
"primary_count": len(primary.get("content", [])),
"secondary_count": len(secondary.get("content", [])),
"total_count": len(merged),
}
@tool(
name="content_retrieval_with_elser_rules_reranker_dev",
description="Perform semantic search using Elasticsearch ELSER model and rerank results",
expected_credentials=[
{"app_id": ES_APP_ID, "type": ConnectionType.KEY_VALUE},
],
)
def content_retrieval_with_elser_rules_reranker(
query: str,
intent: List[str],
size: int = 20,
) -> Dict[str, Any]:
"""
Search Elasticsearch with intent-based chunk filtering and rerank the results.
Args:
query: Natural language search query.
intent: One or more intent labels, such as ["learn", "assess"].
size: Maximum number of results for each search level.
Returns:
Search results and retrieval status.
"""
index_name = DEFAULT_INDEX
model_name = DEFAULT_MODEL
try:
# Get credentials
if is_called_from_orchestrate:
es_username = connections.key_value(ES_APP_ID).get("username", "")
es_password = connections.key_value(ES_APP_ID).get("password", "")
watson_ai_api_key = connections.key_value(WX_APP_ID).get(
"watson_ai_api_key", ""
)
project_id = connections.key_value(WX_APP_ID).get("project_id", "")
else:
es_username = os.getenv("ELASTICSEARCH_USERNAME", "")
es_password = os.getenv("ELASTICSEARCH_PASSWORD", "")
watson_ai_api_key = os.getenv("watson_ai_api_key", "")
project_id = os.getenv("project_id", "")
if not (es_username and es_password):
return {
"status": "Missing ES credentials",
"content": [],
"index_name": index_name
}
if not (watson_ai_api_key and project_id):
return {
"status": "Missing Watson credentials",
"content": [],
"index_name": index_name
}
# Connect to Elasticsearch
es_client = Elasticsearch(
hosts=ES_HOST,
basic_auth=(es_username, es_password),
ca_certs=get_ca_bundle_path(),
verify_certs=True,
request_timeout=60,
)
if not es_client.ping():
return {
"status": "Could not establish Elasticsearch connection",
"content": [],
"index_name": index_name
}
logger.info("Elasticsearch connection successful.")
# Get chunk types for the identified intents
intents_lower = [i.lower() for i in intent]
primary_types = get_chunk_types_by_level(intents_lower, "primary")
secondary_types = get_chunk_types_by_level(intents_lower, "secondary")
logger.info(f"Primary types: {primary_types}")
logger.info(f"Secondary types: {secondary_types}")
# Run primary and secondary searches
empty = {"content": [], "index_name": index_name}
with ThreadPoolExecutor(max_workers=2) as executor:
f_primary = executor.submit(
elser_search_custom,
es_client,
index_name,
query,
model_name,
primary_types,
size,
) if primary_types else None
f_secondary = executor.submit(
elser_search_custom,
es_client,
index_name,
query,
model_name,
secondary_types,
size,
) if secondary_types else None
primary_results = f_primary.result() if f_primary else empty
secondary_results = f_secondary.result() if f_secondary else empty
merged = _merge_results(primary_results, secondary_results)
fallback_used = False
# Search the full index if too few results are returned
if len(merged["content"]) < FALLBACK_THRESHOLD:
logger.warning(
f"Only {len(merged['content'])} intent-filtered results. "
f"Running full-index ELSER search."
)
fallback = elser_search_custom(
es_client,
index_name,
query,
model_name,
chunk_types=None,
size=size,
)
merged_content = fallback
fallback_used = True
else:
merged_content = merged
# Rerank the search results
reranked = rerank_results(
query=query,
contents=merged_content["content"],
watson_ai_api_key=watson_ai_api_key,
project_id=project_id,
top_k=RERANK_TOP_K,
)
logger.info(f"Reranked total: {len(reranked)}")
return {
"content": reranked,
"index_name": merged_content.get("index_name", index_name),
"status": "ok",
"primary_count": merged_content.get("primary_count", 0),
"secondary_count": merged_content.get("secondary_count", 0),
"total_count": len(reranked),
"fallback_used": fallback_used,
}
except Exception as e:
logger.error(repr(e))
return {
"status": f"Unexpected error: {str(e)}",
"content": [],
"index_name": index_name
}
# Run a local test
if __name__ == "__main__":
from dotenv import load_dotenv
load_dotenv()
is_called_from_orchestrate = False
query = "partner ally"
results = content_retrieval_with_elser_rules_reranker(
query,
intent=["learn"]
)
actual = results.content if hasattr(results, "content") else results
print(f"Status : {actual.get('status')}")
print(f"Primary hits : {actual.get('primary_count')}")
print(f"Secondary hits: {actual.get('secondary_count')}")
print(f"Total : {actual.get('total_count')}")
print(f"Fallback used : {actual.get('fallback_used')}")
for item in actual.get("content", []):
print(item["chunk"][:120])
Register the tool with IBM watsonx Orchestrate
Use the orchestrate command-line interface (CLI) to import the retrieval tool into IBM watsonx Orchestrate. The -a option attaches a named connection that provides credentials when the tool runs. You do not need to store credentials in the tool code.
Activate the watsonx Orchestrate environment where you want to register the tool.
orchestrate env activate devOptionally, list the tools that are already available in the active environment.
orchestrate tools listImport the Python tool and its dependencies into the active watsonx Orchestrate environment.
orchestrate tools import \ -k python \ -p search_tool1 \ -f search_tool1/orchestrate_tool.py \ -r search_tool1/requirements.txtThe tool does not attach a connection during import. The
expected_credentialsparameter in the@tooldecorator defines the required connections. The watsonx Orchestrate connections registry provides the credentials when the tool runs.The retrieval tool requires two key-value connections:
es_clientfor Elasticsearch credentials andWatson_apifor watsonx.ai credentials. Add both connections to the active watsonx Orchestrate environment.orchestrate connections add --app-id es_client --type key-value orchestrate connections add --app-id Watson_api --type key-valueProvide the required values when prompted. You can also provide the connection values in a JSON file by using the
--fileoption.Connection Required keys es_clientusername,passwordWatson_apiwatson_ai_api_key,project_idList the available tools to confirm that the import succeeded.
orchestrate tools listConfirm that
content_retrieval_with_elser_rules_reranker_devappears in the list. The retrieval tool is now available to agents in thedevenvironment.
Summary
In this article, you explored a structured framework for improving enterprise search relevance across five stages: Content, Semantic Structuring, Query, Search, and Results. The Content and Semantic Structuring stages prepare documents for retrieval. Content standardization improves content quality, while chunk typing and intent mapping help the search system retrieve content that matches the user's needs. The Query stage improves user queries through techniques such as summarization, rephrasing, HyDE, entity extraction, semantic query expansion, intent recognition, topic filtering, and profanity handling. The Search stage uses techniques such as domain-specific dictionaries, LLM-enhanced relevance training, and vector search to improve document retrieval. The Results stage uses reranking, answer generation, and entity-based filtering to improve the final results.
The five-step pilot methodology uses benchmark results to select, implement, and evaluate retrieval improvements before production deployment.
You can apply the same retrieval framework to customer support, legal search, internal knowledge bases, financial document analysis, and technical documentation. The framework helps search systems connect user queries with relevant content when users and document authors use different terminology.
Acknowledgments
The author (Ridha Juneja) deeply appreciates the support of Sindhoor Bhagwat and Neethu Manuel for the guidance and expertise on reviewing and contributing to this article.