IBM Developer

Tutorial

Accessing IBM Granite LLMs in Jupyter Notebook through Ollama

Experiment with LLMs in your local environment

By Jeya Gandhi Rajan M

On this page

IBM Granite is IBM's family of open, performant, and trusted large language models (LLMs), built to deliver AI-driven insights and capabilities.

For developers or data scientists who are interested in experimenting with the Granite models, Ollama provides a way to run the models locally. By using Jupyter Notebook and Python scripts, you can easily invoke the Granite models through Ollama.

This tutorial walks you through the step-by-step process to access one of the IBM Granite models in a Jupyter Notebook using Ollama. The same setup can also be applied to access other LLMs.

The following diagram shows the architecture of this setup.

arch.png

Prerequisite

You need to install Python 3.x on to your system.

Steps

Step 1. Set up Ollama

Ollama is a tool designed to run and interact with large language models (LLMs) locally on your computer rather than in the cloud. It provides a streamlined interface for deploying and managing models directly on local hardware.

  1. Download and install the Ollama app.

  2. Start the Ollama app.

     ollama serve

  3. Download any Granite model to your system. In this tutorial, I chose granite3-dense:2b.

     ollama pull granite3-dense:2b

  4. Verify that the model downloaded successfully.

     ollama list

    You should get the below output.

     NAME                 ID              SIZE      MODIFIED
 granite3-dense:2b    a9c7deef7ab8    1.6 GB    2 hours ago

Ollama is now ready to serve content from the downloaded LLM.

Step 2. Set up JupyterLab

Next, you need to install JupyterLab in a Python virtual environment.

  1. Go to a folder where you want to install JupyterLab. For example, this is my folder:

     cd /Users/gandhi/GandhiMain/998-work

  2. Create a virtual environment called my-jupyter-env and activate it by using these commands:

     python -m venv my-jupyter-env
 source my-jupyter-env/bin/activate

  3. Install JupyterLab.

     pip install jupyterlab

  4. Run JupyterLab.

     jupyter lab

  5. JupyterLab will start and open in a new browser window at the URL http://localhost:8888/lab/workspaces/auto-U.

Step 3. Create a Jupyter Notebook

Let's create a Jupyter Notebook with a Python script to access the downloaded IBM Granite model.

  1. In the JupyterLab window, click the + button in the top.

    image-11.png

  1. In the Notebook section of the navigator window, choose the "Python 3 (ipykernel)" tile.

    image-12.png

  2. Click the Save button to save the notebook.

    image-13.png

  3. Enter a name for the notebook, then click the Rename button.

    image-14.png

    The notebook is saved in the given name.

    image-15.png

Step 4. Insert scripts into the notebook

Next, you need to update the notebook to access and use the downloaded LLM.

  1. Create four separate cells in the notebook. Then, copy these four scripts into those four separate cells.

    Cell 1:

     !pip install langchain-ollama

    Cell 2:

     model_id = "granite3-dense:2b"

    Cell 3:

     from langchain_ollama.llms import OllamaLLM
 model = OllamaLLM(model=model_id)

    Cell 4:

     prompt = "What is Kubernetes"
 response = model.invoke(prompt)
 print(response)

  2. Verify that your notebook looks like this.

    image-21.png

These scripts, including MD content surrounding the them, are available in the sample.ipynb notebook here. You can copy and paste this into your notebook and execute it.

image-29.png

Step 5. Execute the Jupyter Notebook

Let's execute the notebook, to see the IBM Granite model in action.

  1. Ensure the kernel is pointing to Python 3 (ipykernel) in the top right corner.

    image-22.png

  2. Run all four cells to see the output.

    image-28.png

We have successfully executed the notebook and called the IBM Granite model via ollama.

Step 6. Close the apps in our local environment

  1. First, close JupyterLab by choosing File > Shutdown from the menu.

    image-30.png

  2. Deactivate the virtual environment.

     deactivate

  3. Go to the terminal window where Ollama is running, and press Ctrl+C.

Conclusion

As demonstrated in this tutorial, by using Ollama with JupyterLab, you can experiment with the LLMs in your local environment.