Build an AI research agent for image analysis with Granite

In this tutorial, you'll build an AI research agent that conducts in-depth research based on image analysis. Learn how to combine Granite 3.2 vision and language models, CrewAI, and RAG to convert images into comprehensive research reports locally.

Using the Granite 3.2 Vision model alongside the Granite 3.2 8B language model, you'll learn how to create an advanced image researcher. You can run everything locally using Ollama, Open WebUI, and Granite models, ensuring a private and cost-effective solution.

This tutorial uses CrewAI as the agentic AI framework to orchestrate parallel, asynchronous research tasks. This approach enables efficient exploration of complex visuals, transforming images into actionable insights. Additionally, CrewAI builds on Langchain, an open-source project that provides tools for agents.

To power the research, this tutorial incorporates retrieval-augmented generation (RAG), enabling the agent to fetch relevant information from both web sources and user-provided documents. This ensures that the generated insights are accurate and grounded in real-time content.

Building on our previous work

In our previous tutorial, Build a multi-agent RAG system with Granite locally, we demonstrated how to build an agent that constructs a sequential plan to accomplish a goal. In this tutorial, we'll explore alternative methods of agent collaboration by developing a research agent that identifies a structured plan of researchable topics from an image. It then commissions parallel research agents to investigate each topic using RAG and synthesizes findings into a comprehensive report.

You can access the open source implementation of this agent and setup instructions in the Granite Retrieval Agent GitHub repository.

The sample application: Turning images into knowledge

This Image Research Agent transforms images into actionable insights by filling gaps in your understanding and offering deeper context. Whether the image is a complex technical diagram or a historical photograph, the agent can break down its components and provide relevant details on every element.

This Image Research Agent supports these use cases:

• Architecture diagrams: Understand components, protocols, and system relationships.

• Business dashboards: Explain KPIs, metrics, and trends in BI tools.

• Artwork and historical photos: Analyze artistic styles, historical context, and related works.

• Scientific visualizations: Interpret complex charts, lab results, or datasets.

By combining vision models, agentic workflows, and RAG-based research, this solution empowers you to transform visual data into meaningful insights that support informed decision-making, deeper learning, and enhanced understanding across industries.

How the AI agents work together

The following image shows the workflow in action:

Environment setup

To set up your local environment, follow the steps outlined in the IBM Granite Multi-Agent RAG tutorial and the Granite Retrieval Agent GitHub repository.

This solution uses OpenWebUI as a user interface and Ollama for local inferencing, ensuring privacy and efficiency. Both components are powered by open-source tools, making it easy to deploy and manage the entire workflow on your local machine.

Steps

Step 1. Set up connections to the two large language models (LLMs)

First, set up connections to the two LLMs:

• Granite Vision LLM for transforming images into descriptive text

• Granite 3.2 Instruct for all other language-related tasks including planning and reasoning.

This tutorial uses the q8_0 quantization instead of the default q4_K_M. This quantization provides better tool-calling reliability while still being lightweight enough to run locally.

llm = LLM(
    model="granite3.2:8b-instruct-q8_0",
    base_url="http://localhost:11434",
    api_key="ollama",
    temperature=0
)

vision_llm = LLM(
    model="granite3.2-vision:2b",
    base_url="http://localhost:11434",
    api_key="ollama",
    temperature=0
)

Step 2. Define the Research Item Identifier agent

This agent analyzes the image description and user query to identify key features and concepts requiring research.

Inputs:

• Image description generated by the Image Explainer.
• User's research goal or context.

Outputs:

• Structured list of items for further research.

class ResearchItem(BaseModel):
    item_name: str
    research_instructions: str

class ResearchItems(BaseModel):
    items: List[ResearchItem]

item_identifier = Agent(
    role='Item Identifier',
    goal='Identify concepts in the image requiring research.',
    llm=llm
)

identification_task = Task(
    description='Identify all researchable items in the image description.',
    agent=item_identifier,
    expected_output='List of items and concepts for further research.',
    output_pydantic=ResearchItems
)

identifier_crew = Crew(
    agents=[item_identifier],
    tasks=[identification_task],
    process=Process.sequential,
    verbose=True
)

We are using a crew that consists of a single task and a single agent. The output_pydantic=ResearchItems portion instructs the LLM to output its structure into a Python object that can be directly leveraged in the next step of the plan.

Step 3. Define the Research Crew

For each identified research item, a separate Research agent runs asynchronously, using RAG to search the web and user documents. Because the output of the previous agent is a Python list object, you can provide that list of research items to the Research Crew to kick off parallel, asynchronous research tasks for each item in the list.

Inputs:

• List of research items from the Research Item Identifier.

Outputs:

• Detailed information, including references and explanations, for each item.

researcher = Agent(
    role='Researcher',
    goal='Conduct detailed research for each identified concept.',
    tools=[do_web_search, do_knowledge_search],
    llm=llm
)

research_task = Task(
    description='Fulfill the instruction given. {item_name} {research_instructions}',
    agent=researcher,
    expected_output='Information that directly answers the instruction given, with in-line citations for every reference.'
)

research_crew = Crew(
    agents=[researcher],
    tasks=[research_task],
    verbose=True
)

Step 4. Pull the agents together

Make a call to the Granite Vision model to describe the image:

image_query = "Describe the image in detail for educational purposes."
image_description = vision_llm.call([
    {"role": "user", "content": image_query, 'type': 'image_url', 'image_url': {'url': 'data:image/png;base64,.....'}}
])

Use the Research Item Identifier Crew to create a structured Python object representing all of the items in the image that need to be researched:

identifier_crew.kickoff(image_description)

Then, use an asynchronous, parallel kickoff of the “Research Crew” to perform retrieval for each research query:

tasks = []
for task in identification_task.output.pydantic.items:
    tasks.append({"item_name": task.item_name, "research_instructions": task.research_instructions})

output = research_crew.kickoff_for_each_async(tasks)

Finally, summarize the findings into a comprehensive report:

final_output = llm.call(f"""Thoroughly answer the user's question, citing all references.
User query: {instruction}
Image description: {image_description}
Gathered information: {output}""")

Example: Analyzing a sample image

The following example uses a sample image of the Statue of Liberty, a public domain image courtesy of the Library of Congress, specifically a photo taken by Carol M. Highsmith. The goal is to learn about the history of the statue and anything else depicted in the image.

The first step of the agent's process is describing the image, and then breaking that down into individual items to be researched. Below is the output of the agent, which, when deployed in OpenWebUI, can be seen in the Open WebUI server logs.

Once the tasks have been established, a swarm of Research Agents begin their tasks. The following image shows some of the task output.

After all the researchers complete their task, you see a detailed output report, complete with hyperlinks to all referenced sources.

Conclusion

The Image Researcher Agent demonstrates how generative AI can transform static images into rich, educational insights. By using CrewAI, the Granite Vision and Reasoning models, and parallel research workflows powered by RAG, you can explore complex visuals with clarity and context.

The capabilities built on top of transformer-based large language models have come a long way. With the advent of agentic AI, models can now function as research tools capable of driving automation and deep research tasks. Agentic AI offers several solutions for enterprise use cases, such as analyzing merchandise pricing for marketing research, studying scientific research papers for discovery and understanding, and even optimizing hiring processes by gathering and reporting on resumes and LinkedIn profiles.

Try deploying your own Image Researcher Agent using the code provided to explore the potential of image-driven learning.

Next steps

Explore this demo project in the Granite Retrieval Agent GitHub repository.

Try out Granite in the IBM Granite Playground.

Read more in the IBM Granite documentation.

Acknowledgements

A tremendous thank you to Vanna Winland for her refinement and editing of this article's content.