Introducing OpenRAG on watsonx.data

TL;DR: OpenRAG on watsonx.data is IBM's out-of-the-box managed agentic RAG service that prepares enterprise content, retrieves the right evidence, and orchestrates agentic workflows, all within enterprise security and governance controls. This blog explains how it works, how it differs from traditional RAG, and how to get started.

Today’s enterprises identify inaccurate AI responses as one of the top 3 barriers to adopting AI. This perspective exposes a critical gap between an AI system that can generate an answer and one that an enterprise can trust. Often, the problem is not the language model itself. It is the context that is supplied to the model, which is incomplete, stale, poorly parsed, retrieved with the wrong search strategy, or disconnected from enterprise access controls.

Reliable generative AI requires more than a capable model. It requires a system that can prepare complex enterprise content, retrieve the right evidence for each task, adapt when the first search is insufficient, and operate within enterprise governance controls.

To solve this issue, IBM recently announced OpenRAG on watsonx.data, an out-of-the-box managed RAG service. OpenRAG on watsonx.data is built on the OpenRAG open-source project. This open foundation gives developers visibility into the underlying architecture and access to its core technologies: Docling for document processing, OpenSearch for retrieval, and Langflow for agent orchestration.

• Docling transforms complex documents into AI-ready content while preserving structure and context across text, tables, images, headings, and other document elements. It also provides configurable chunking strategies for different content types and use cases.
• OpenSearch provides scalable lexical, semantic, and hybrid retrieval. By combining conceptual similarity with exact keyword matching and metadata filtering, it helps retrieve evidence that is relevant to both the meaning and the specific details of a question.
• Langflow provides visual orchestration for building and customizing agentic retrieval workflows. It defines how an agent interprets a request, retrieves information, assesses the available context, and uses additional searches or tools when needed.

Benefits of OpenRAG on watsonx.data

OpenRAG on watsonx.data brings together the capabilities that are needed across the RAG lifecycle: connecting distributed enterprise knowledge, processing complex content, indexing it for search, retrieving relevant evidence, and orchestrating how an AI agent uses that evidence.

OpenRAG on watsonx.data moves beyond the fixed, one-shot retrieval pattern used by many traditional RAG systems. Instead of submitting one query and generating an answer from the first results, OpenRAG on watsonx.data:

• Interprets the user’s intent and conversation context
• Determines whether enterprise knowledge is required
• Formulates or refines the search query
• Retrieves and assesses the available evidence
• Searches again or uses another tool when the evidence is insufficient
• Validates the assembled context before generating a grounded response

This adaptive approach is particularly valuable when a question is ambiguous, spans multiple sources, requires several reasoning steps, or cannot be resolved by a single retrieval pass.

How OpenRAG on watsonx.data is different from traditional RAG

Traditional RAG typically follows a fixed pattern: convert the question into an embedding, run one search, retrieve the highest-ranked chunks, and pass them to a large language model (LLM). (Learn more about RAG in the IBM RAG Cookbook.)

OpenRAG on watsonx.data extends this pattern with structure-aware document processing, hybrid retrieval, agentic orchestration, and a managed enterprise foundation:

• Adapt retrieval to the task. A traditional RAG pipeline generally applies the same retrieval sequence to every question. OpenRAG makes retrieval part of the agent’s reasoning process. The agent can interpret the user’s intent, determine whether enterprise retrieval is needed, select an appropriate retrieval strategy, and formulate or refine the query. If the initial search does not provide sufficient evidence, the agent can search again or use another available tool. This iterative approach helps assemble more complete and relevant context for complex, ambiguous, or multistep questions before asking the LLM to generate a response.

• Retrieve using multiple relevance signals. Traditional RAG implementations commonly rely on vector similarity alone. OpenRAG uses OpenSearch to support lexical, semantic, and hybrid retrieval. This allows applications to combine conceptual similarity with exact keyword matches and metadata filters. The combined approach is important for enterprise questions that depend on both meaning and precision, such as queries containing product names, error codes, policy clauses, dates, or other specific identifiers.

• Preserve meaning when preparing enterprise content. Retrieval quality depends on how source content is processed before it is indexed. OpenRAG uses Docling to process complex enterprise documents while preserving important structure and context across elements such as headings, tables, images, and reading order. OpenRAG can ingest content from enterprise repositories and common formats, including PDFs, presentations, spreadsheets, images, and text documents. The content is processed, chunked, embedded, and indexed as searchable, AI-ready knowledge for assistants, enterprise search, customer support, sales, procurement, and other grounded AI applications.

• Reduce integration and operational effort. Building a production RAG system requires more than connecting a vector database to an LLM. Teams must integrate and operate document processing, chunking, embedding generation, indexing, retrieval, agent orchestration, service interfaces, security, and infrastructure. OpenRAG on watsonx.data packages ingestion, indexing, retrieval, orchestration, and application interfaces into an out-of-the-box managed service. OpenSearch is provisioned as the required retrieval backend, while Langflow provides visual orchestration for inspecting and customizing ingestion and agent workflows. Teams can begin with built-in experiences and extend them as requirements evolve.

• Add an enterprise foundation. A working RAG prototype does not automatically provide the controls required for enterprise deployment. OpenRAG on watsonx.data adds managed infrastructure, scaling and operations, role-based access control, IBM identity and billing integration, and enterprise security and governance capabilities.

The difference is therefore not simply how OpenRAG retrieves a chunk. It is how the complete system prepares knowledge, adapts retrieval to the task, iterates when evidence is incomplete, uses tools when needed, and delivers governed enterprise context across a growing number of users and applications.

Architecture of OpenRAG on watsonx.data

Applications and developers access OpenRAG on watsonx.data through the web interface, SDKs, REST APIs, or the built-in Model Context Protocol (MCP) server. The OpenRAG backend coordinates the underlying services, manages configuration, and runs ingestion and query workflows. Authentication, role-based access control, and operational controls are applied through the service layer.

OpenRAG on watsonx.data uses two primary runtime paths:

• A knowledge ingestion path that prepares and indexes enterprise content
• An agentic query path that retrieves and uses that content to generate responses

Knowledge ingestion path

OpenRAG transforms enterprise content into an AI-ready, searchable knowledge base through five stages: ingestion, document processing, content preparation, embedding generation, and indexing.

1. Connect and ingest knowledge. Content can be added through direct file and folder uploads, URLs, and connectors for sources such as Amazon S3, Google Drive, Microsoft OneDrive, and Microsoft SharePoint. Ingestion runs as a background task through the built-in OpenSearch Ingestion flow in Langflow.
2. Process content with Docling. The ingestion flow sends supported files to Docling Serve. Docling parses the content and performs operations such as OCR, table extraction, and structure-aware document processing. It produces a DoclingDocument containing the extracted content and associated metadata.
3. Prepare and chunk the content. The processed document is exported into a standardized Markdown representation. The flow enriches it with metadata such as the filename, file type, file size, connector information, and document ownership. It then divides the content into chunks using the configured chunk-size and overlap settings.
4. Generate embeddings. Each chunk is sent to the configured embedding model, which converts it into a vector representation. OpenRAG supports configurable embedding providers, allowing teams to select a model that fits their deployment and retrieval requirements.
5. Index the knowledge in OpenSearch. OpenSearch stores the content chunks, vector embeddings, and associated metadata. The resulting knowledge base supports lexical, semantic, and hybrid retrieval, enabling applications to combine exact term matching with meaning-based search and metadata filters.

At the end of ingestion, complex enterprise content has been transformed into an indexed knowledge base that can be searched by the agentic retrieval workflow.

Agentic query and response path

OpenRAG handles each request through an agentic workflow that interprets intent, retrieves and evaluates evidence, and iterates when needed before generating a grounded response.

1. Receive the request. A user or application submits a request through OpenRAG Chat, an SDK, a REST API, or MCP. The flow can also receive relevant conversation context and knowledge filters.
2. Interpret the request. The Langflow agent uses the configured language model to interpret the user's intent. It determines whether enterprise retrieval is required and decides which available tool can help answer the request.
3. Retrieve enterprise evidence. When the agent selects OpenSearch, the configured embedding model generates a vector representation of the query. OpenRAG applies any relevant knowledge filters and searches the OpenSearch knowledge base for supporting content.
4. Assess and refine the context. The agent evaluates the retrieved information instead of automatically accepting the first result set. If the evidence is insufficient, it can refine the query, search OpenSearch again, or call another available MCP tool.
5. Generate a grounded response. Once sufficient context has been assembled, the language model uses the retrieved enterprise evidence to generate a grounded response. The response is then returned to the calling application through the same service interface.

This iterative execution pattern enables OpenRAG to handle questions that are ambiguous, span multiple sources, or require more than one retrieval step.

Open source by design, enterprise-ready on watsonx.data

The OpenRAG open-source project gives individual developers and technical teams an accessible way to explore agentic RAG, experiment with retrieval workflows, customize components, and run the stack in an environment they manage. It provides direct access to capabilities including document ingestion, hybrid retrieval, configurable models, Langflow workflows, iterative search, and tool use.

As OpenRAG moves from individual experimentation to enterprise deployment, however, the requirements extend beyond core RAG capabilities. Supporting enterprise data, applications, and users also requires teams to deploy and integrate the components, secure access, monitor services, manage infrastructure, troubleshoot failures, apply updates, and scale capacity as adoption grows.

OpenRAG on watsonx.data addresses these requirements by delivering the open, agentic foundation as an out-of-the-box managed service. It packages ingestion, indexing, retrieval, and orchestration into an integrated experience while adding managed infrastructure, scaling, security, governance, compliance, workload isolation, role-based access controls, and operational visibility.

This managed approach can also help lower the total cost of ownership of enterprise RAG. Although open-source components are available without software licensing costs, organizations must still account for the people, infrastructure, and operational processes required to integrate, secure, maintain, troubleshoot, and scale the complete system throughout its lifecycle.

By reducing this build-and-operate burden, OpenRAG on watsonx.data enables teams to direct more resources toward improving retrieval quality, developing AI applications, and delivering business value. OpenRAG OSS provides the open and accessible foundation for developer experimentation, while OpenRAG on watsonx.data operationalizes that foundation for secure, governed, and scalable enterprise adoption.

Getting started with OpenRAG on watsonx.data

OpenRAG is available with IBM watsonx.data Premium, providing a managed path from fragmented enterprise content to agentic applications grounded in relevant business context. It brings document preparation, hybrid retrieval, agent orchestration, and enterprise controls into an integrated experience. Learn more in the watsonx.data documentation.

Ready to see it in action? Book a live demo of OpenRAG on watsonx.data to explore how it prepares, indexes, and retrieves enterprise knowledge for grounded AI responses.