IBM Developer

Tutorial

Use IBM Granite LLMs in watsonx.ai flows engine

Learn how to use IBM Granite LLMs in watsonx.ai flows engine for basic prompting, summarization, and classification tasks

By Roy Derks

On this page

New models are released almost every day, making it hard for developers to know which one to use based on how they differentiate or are priced. That’s why having the flexibility to experiment with various models can significantly enhance the development and deployment of AI applications. Watsonx.ai flows engine offers this flexibility by providing a unified API that works seamlessly with all models available on the IBM watsonx platform.

This is the second tutorial in a series of four, in which you learn how to work with different large language models (LLM) in IBM watsonx.ai flows engine. Experimenting with models that are available in watsonx.ai (like IBM Granite, Meta’s Llama, or Mistral) is completly free and doesn’t require a credit card. After completing the entire series, you'll be able to build generative AI flows that are tailored to each of these models, and deploy them to the cloud using watsonx.ai flows engine.

Prerequisites

To follow this tutorial, you must have watsonx.ai flows engine set up, as described in the Using different LLMs in watsonx.ai flows engine tutorial.

Using IBM Granite in watsonx.ai flows engine

In this specific tutorial, you learn to use LLMs from the IBM Granite series, which are a series of open source models created by IBM. The IBM Granite LLMs are foundation models designed to be open, trusted, and targeted, empowering both enterprises and developers. These models are optimized for various applications, offering high performance and flexibility for tasks like text completion, summarization, and translation. You could also use Granite for more complex tasks like code generation or to support Retrieval Augmented Generation (RAG), as you see in the Build a RAG application with watsonx.ai flows engine tutorial.

The series consists of four foundation models:

  • Granite 13B chat: Works best for most general use cases because it's optimized for conversational tasks

  • Granite 13B instruct: LLM trained specifically on finance data to help with tasks in the financial space

  • Granite multilingual: Multilingual LLM trained on English, German, Spanish, French, and Portuguese

  • Granite Japanese: Designed to perform language tasks on Japanese text

Let's look at three use cases (basic prompting, summarization, and classification) using watsonx.ai flows engine IBM Granite. You’ll primarily be using using the chat and instruct model, but you could swap these for the multilingual models if you want to try the following use cases in your native language.

Creating different generative AI flows

With watsonx.ai flows engine, you can create flows for many different generative AI use cases. Prompting is an important part of each of these use cases, and most LLMs have a specific prompt template that you should use to get the best possible results. The format of the prompt template is often linked to how the LLM was trained. The most versatile IBM Granite model is granite-13b-chat-v2, which is optimized for conversational tasks, work with RAG, and natural language processing (NLP) tasks such as summarization or classification. You can find the prompt template and more prompting tips for the IBM Granite models in the Prompting the granite-13b-chat-v2 foundation model from IBM documentation.

Basic prompting

In the first example, you use the granite-13b-chat-v2 model to answer a question. The base prompt for this LLM looks like the following example.

<|system|>
You are Granite Chat, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior. You always respond to greetings (for example, hi, hello, g'day, morning, afternoon, evening, night, what's up, nice to meet you, sup) with "Hello! I am Granite Chat, created by IBM. How can I help you today?". Please do not say anything else and do not start a conversation.
<|user|>
{question}
<|assistant|>

In the previous code, the {question} variable should be subsituted with the question you want the LLM to answer. To use this prompt template in watsonx.ai flows engine, you can add it to a flow in the wxflows.toml file. Assume that you have a textCompletion flow that consists of the promptTemplate and completion steps. The promptTemplate would be used to define the prompt template.

flows="""
    textCompletion = templatedPrompt(promptTemplate: "{question}") | completion(model: textCompletion.model, parameters: textCompletion.parameters)
"""

The templatedPrompt step should include the base prompt, and is able to subsitute the {question} variable to the argument that you pass when consuming the textCompletion flow. To add the prompt template, you can copy-paste the following code into your wxflows.toml file.

flows="""
    textCompletion = templatedPrompt(promptTemplate: "<|system|>You are Granite Chat, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior. You always respond to greetings (for example, hi, hello, g'day, morning, afternoon, evening, night, what's up, nice to meet you, sup) with 'Hello! I am Granite Chat, created by IBM. How can I help you today?'. Please do not say anything else and do not start a conversation.<|user|{question}<|assistant|>") | completion(model: textCompletion.model, parameters: textCompletion.parameters)
"""

Alternatively, you can use the graniteChatPrompt step, which already includes the base prompt template.

After adding this flow, you must deploy the flow to your flows engine endpoint by running the following code.

wxflows deploy

The endpoint that the flow is deployed to is printed in your terminal, and you can use it to send a request by using the SDK for JavaScript or Python. The previous tutorial showed how to set up watsonx.ai flows engine and how to use the JavaScript SDK. To use the updated textCompletion flow with this example, use the following JavaScript snippet in the app/index.js file.


const flowName = 'textCompletion'
const question = Who is the president of France?

const result = await model.flow({
    schema,
    flowName,
    variables: {
        question,
        model: "ibm/granite-13b-chat-v2",
        parameters: {
            stop_sequences: []
        },
    },
})

To run this file, you can execute node index.js from your command line. The question "Who is the president of France?" is injected into the base prompt, and the answer is printed in your terminal.

Besides basic prompting, you can use the same prompt template with a few minor adjustments to create AI flows for summarization and classification.

Summarization flow

One of the most popular use cases for LLMs is summarizing large pieces of text. Previously, I published a tutorial on building a web scraping tool where watsonx.ai flows engine was used to summarize the contents of a web page. That tutorial included a summarization flow that you must add to the wxflows.toml file in your project.

flows="""
    summarization = templatedPrompt(promptTemplate: "<|system|>You are Granite Chat, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior.<|user|>Create a concise summary of the document provided below. [Document]{question}[End]<|assistant|>") | completion(completion(model: summarization.model, parameters: summarization.parameters)
"""

This flow has the same base prompt template that you used in the previous section, but this time it includes the [Document] and [End] tags to indicate the beginning and end of a document that's included in the prompt template. These tags help IBM Granite to understand what content should be summarized and distinguish it from your question or instructions in the prompt.

To summarize a document, you can add the following JavaScript code to the app/index.js file.

const flowName = 'summarization'
const question = 'INSERT YOUR TEXT HERE'

const result = await model.flow({
    schema,
    flowName,
    variables: {
        question,
        model: "ibm/granite-13b-chat-v2",
        parameters: {
            stop_sequences: []
        },
    },
})

Insert a text to summarize. For example, copy-paste the contents from this tutorial.

In the summarization flow, the model can be set dynamically, but it can also be defined in the flow itself. To define the model inside the flow, you can make the following change.

flows="""
    summarization = templatedPrompt(promptTemplate: "<|system|>You are Granite Chat, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior.<|user|>Create a concise summary of the document provided below. [Document]{question}[End]<|assistant|>") | completion(model: "ibm/granite-13b-chat-v2" parameters: summarization.parameters)
"""

Now, every time you send a request to flows engine that includes the summarization flow, it automatically uses the model ibm/granite-13b-chat-v2 for completion. You no longer need to set the model when using the SDK for this flow.

const flowName = 'summarization'
const question = 'INSERT YOUR TEXT HERE'

const result = await model.flow({
    schema,
    flowName,
    variables: {
        question,
        parameters: {
            stop_sequences: []
        },
    },
})

Defining the model inside the flow makes the most sense when you're also defining the prompt template in the flow, as you've learned that LLMs require a specific formatting of a prompt to best serve the user.

Classification flow

The final generative AI use case that you'll be looking at is classification. LLMs can be used for classification, but these NLP tasks can also be done effectively by smaller models. To use IBM Granite for classification, you'll use a few-shot prompting technique where you present the LLM with a couple of examples and have it fill in the last example. The LLM classifies whether a comment is positive or negative.

First, you must create a new flow called classification in the wxflows.toml file that looks like the following example.

flows="""
    classification = templatedPrompt(promptTemplate: "<|system|>You are Granite Chat, an AI language model developed by IBM. You are a cautious assistant. You carefully follow instructions. You are helpful and harmless and you follow ethical guidelines and promote positive behavior.<|user|>For each feedback, specify whether the content is Positive or Negative. Your response should only include the answer. Do not provide any further explanation. Here are some examples, complete the last one: {question}<|assistant|>") | completion(model: classification.model, parameters: classification.parameters)
"""

You didn't define the task and model in the flow. Instead, these are defined when flows engine is called with a flow. However, before you can call the endpoint, you must deploy this new flow by running the following code.

wxflows deploy

This code makes the classification flow available so that it can be called using the SDK by changing the code in app/index.js.

const flowName = 'classification'
const question = `
    Feedback: John Doe is the best developer I've ever seen.
    Class: Positive

    Feedback: Jane Smith did a bad job in our team
    Class: Negative

    Feedback: Foo Bar has made a great impression on all of us
    Class:
`

const result = await model.flow({
    schema,
    flowName,
    variables: {
        question,
        model: "ibm/granite-13b-chat-v2",
        parameters: {
            decoding_method: 'greedy',
            stop_sequences: []
        },
    },
})

The LLM is tasked with classifying the last comment, and after running node index.js, it should print the classification (which should be "Positive") in your terminal.

When you change the model from ibm/granite-13b-chat-v2 to ibm/granite-13b-instruct-v2, you see that the last comment is still classified, but the response is less "chatty." Also, by setting the decoding_method parameter to "greedy," only the options that are most probable are chosen, which is recommended when using an LLM for classiying text.

You now have three flows deployed that can be used for separate use cases. The complete code for this tutorial can be found on GitHub.

What's next?

In this second tutorial of the series, you learned how to use IBM Granite LLMs in watsonx.ai flows engine for basic prompting, summarization, and classification tasks. By following the steps outlined in this tutorial, you can build generative AI flows that are tailored to each of these models, and compare different models available in watsonx.ai. The next two tutorials in this series will cover using watsonx.ai flows engine with Meta's Llama as well as Mistral models.

If you have any questions or feedback, head over to our Discord community and let us know!