Build a local RAG system with NVIDIA DGX Spark and OpenRAG

In this tutorial, we’re going to turn an NVIDIA DGX Spark into a powerful local RAG system using OpenRAG, an open-source agentic RAG framework. The goal is to build a private system that can ingest your documents, retrieve the right context, and let you chat with your knowledge locally on a tiny, but absurdly powerful, machine. All with just a few prompts.

Before we start, it’s worth understanding why the DGX Spark is such a remarkable piece of hardware. At a high level, NVIDIA DGX Spark delivers one petaFLOP of AI performance in a box about the size of a Mac mini. A FLOP is how many simple mathematical computations (add/subtract/multiply/divide) a computer can execute in one second. A petaFLOP is one quadrillion of them. For perspective: the first petaFLOP-capable computer, IBM Roadrunner in 2008, filled 296 racks, took up 6,000 square feet, used 2.35 megawatts of power, and cost about $100 million. What required massive resources not too long ago, can now sit on your desk for about $4,000.

While we’re using DGX Spark in this tutorial to run a fully local, highly capable setup, OpenRAG can run on any laptop you use every day and does not require a dedicated GPU. It also supports a wide range of models, but we’ll keep everything local by using Ollama to host NVIDIA Nemotron 3 Super as the LLM and EmbeddingGemma for embeddings. As a system, OpenRAG can decide when to search, call the right tools, retrieve additional context, and refine its response iteratively until it has enough information to provide a grounded answer.

OpenRAG packages together:

• Langflow – for agent and data‑flow orchestration.
• OpenSearch – for indexing and queries (the R in RAG).
• Docling – for state‑of‑the‑art document and content extraction.
• OpenRAG Application Layer – to pull all the application parts together.
• Terminal UI (TUI) - for setup and monitoring.

Step 1: Get an AI coding agent

You’ll want an AI coding agent that can drive your terminal.

My favorite is Warp, but you can use alternatives such as Claude Code, Cursor, or our very own IBM Bob.

• Sign up for the free trial. (You can use my referral link so I can get a t-shirt.)
• Install Warp either:
  • Directly on the Spark
  • On your local machine and then shell into the Spark

I usually do the latter.

In Warp, you can choose your model. For this tutorial, I suggest GPT-5.4 with reasoning effort set to high but "auto" and other models I tested also worked.

Step 2: Install Ollama (the model server)

We’ll use Ollama because it’s fast and incredibly easy.

In Warp on the Spark, issue the following prompt.

Prompt:
Install ollama and start it as a system service.

Step 3: Load models into Ollama

Now we’ll load and test our models.

Prompt:
Load the latest version of nemotron‑3‑super into ollama and test it. Prefer NVFP4 quantization if it is available.

Prompt:
Install the latest embeddinggemma in ollama and test it.

Prompt:
Install the uv package manager so Python tools can be executed in isolated environments.

Prompt:
Use uv to create a new project and virtual environment using Python 3.13. Call the project . Add openrag to that project. OpenRAG is available via uv pip.

Change my_project to whatever name you like.

A quick warning:
Don’t completely YOLO these prompts.

For example, when I asked for embeddinggemma, the agent initially pulled Gemma instead. I had to correct it.

Step 5: Start OpenRAG

Let’s get OpenRAG up and running.

Prompt:
Start openrag in my_project. Start OpenRAG using the terminal user interface (TUI) within the project environment using the --tui flag.

You should now be dropped into the OpenRAG TUI.

Step 6: Set up OpenRAG

From the TUI:

1. Follow the Basic Setup path.
2. Choose a password that meets the complexity requirements. Write it down.
3. Save and start the services.

OpenRAG will download and start all required containers. This can take a while on slower connections — be patient.

Once everything is running, click the Status button.

You should see: Mode: GPU in the top corner. We’re accelerated!

Step 7: Access the Web UI

In the status screen, note the openrag-frontend entry. It will show an IP address and port (usually 0.0.0.0:3000).

• If you’re on the Spark desktop: Open http://localhost:3000

• If you’re SSH’d in from another machine: Use http://<spark-ip>:3000

This brings up the OpenRAG Web UI.

Step 8: Configure the agent

When you set up the agent:

1. Choose Ollama as the provider.
2. Confirm the Base URL (should already be correct).
3. Select Nemotron‑3‑Super from the model drop-down.

If the model isn’t already loaded, it might take approximately 30 seconds to pull it into memory.

The UI will then guide you through ingesting and chatting with your first document, a PDF file, Word document, or anything else.

Step 9: Explore the Langflow logic

Finally, let’s look under the hood.

In OpenRAG, navigate to: Settings > Agent > Edit in Langflow.

This opens the Langflow canvas where the agent’s logic is defined.

From here you can explore the other flows which power OpenRAG such as:

• The main agent flow
• Additional flows for ingesting files and URLs
• How retrieval, prompting, and responses are wired together

Use the OpenRAG UI to ingest more documents in the Knowledge section. OpenRAG makes it easy to connect AI workflows to knowledge bases, whether they are stored locally or in cloud storage systems like Google Drive, Microsoft OneDrive, or AWS S3. OpenRAG supports more than 20 file types including PDFs, images, PowerPoints and Excel files. When you ask the agent any questions it will always check whatever knowledge you’ve provided.

All of this is offline, running on your private little PetaFLOP AI machine. But you might be wondering, “Could I run this on my current computer?” OpenRAG can run on just CPU and use popular model hosting providers. But when it comes to running models locally, you want a good GPU. How good? Well, that’s a function of a couple of things which highlight the value of the way this machine was designed.

The Nemotron 3 family of models uses a hybrid Transformer–Mamba architecture, which reduces inference-time memory pressure in both capacity and bandwidth. The IBM Granite 4.0 family of models uses a similar architecture. But if you want to run larger models (which is where the real intelligence is), you need more memory than most gaming graphics cards. The NVIDIA RTX 5090 (currently the best gaming GPU on the market) has 32GB VRAM. The NVIDIA DGX Spark has 128GB memory shared between its Grace CPU and Blackwell GPU. With that I can run much larger models, such as gpt-oss-120B and Nemotron 3 Super with NVFP4 quantization, and still have enough RAM to run an embedding model and a few other things!

In the previous screen capture, I’m running gpt-oss:120b, because Ollama cannot yet run NVFP4. If you use the Q4_K_M variant of Nemotron 3 Super available to Ollama, it will consume all of the available memory. There are NVFP4-optimized versions of Nemotron 3 Super available if you are running vLLM but it’s a bit of a science project to get up and running correctly. By all means, partner with your AI to see if there is a formal NVIDIA NIM available yet or if ollama has been upgraded to support NVFP4.

Summary

In our opinion, the future of how we run AI will be a hybrid Edge + Cloud model. The software we use will orchestrate a mixture of models and tools on our local (aka “Edge”) devices like the NVIDIA DGX Spark, and models and tools in the cloud.

OpenRAG on NVIDIA DGX Spark shows just how fast and capable the “edge” side of that experience can be. As for the hybrid Edge + Cloud vision… OpenRAG can do that today.