Using Meta Llama 3.1 in watsonx.ai flows engine

With the rapid release of new large language models (LLM), it can be challenging for developers to select the most suitable model based on differentiation or pricing. The flexibility to test various models before implementing AI into your application is crucial for the development and deployment of AI apps. With IBM watsonx.ai flows engine, you can experiment with different models, like Meta's Llama 3.1, by using a set of developer tools such as a CLI and SDK. You can sign up for the free plan today, and get started building AI applications for free.

This tutorial is part of a four-part series that guides you through working with different LLMs like Meta's Llama 3 within IBM watsonx.ai flows engine. You'll also learn how to optimize your prompts and flows to work best for text completion and code generation, and how to best utilize the large context window that's offered by Llama 3.1.

Prerequisites

To follow this tutorial, you must have watsonx.ai flows engine set up, as described in the Using different LLMs in watsonx.ai flows engine tutorial.

Using Meta Llama 3.1 in watsonx.ai flows engine

In this tutorial, learn how to implement and leverage the capabilities of Llama 3.1, Meta's latest release in a series of open source LLMs. The Llama models are engineered to provide robust and reliable performance across a range of tasks including text completion, summarization and code generation, and more advanced tasks like tool calling and retrieval augmented generation (RAG).

The Llama 3.1 lineup consists of the following foundation models:

• Llama 3.1: This model is finely tuned for general conversational tasks, making it ideal for a broad array of use cases. The model has three variations in parameters: 8B, 70B, and 405B.
• Llama 3.1 Instruct: Use this model when you want to combine conversation with tool calling or function calling. It is not recommended when you want to solely do conversational tasks. The model also comes with three variations in parameters: 8B, 70B, and 405B.
• Llama Guard 3: An LLM with built-in guardrails that can be applied to both the input and output of the LLM.

All of the Llama 3 and 3.1 models are multilingual, although in this tutorial you use English only. You're going to compare a couple of different tasks, learn how to use the Meta Llama 3.1 models in watsonx.ai flows engine, and optimize the prompt template for each of these use cases.

Text completion

The first use case you explore is "simple" text completion, which means you ask the LLM a natural language question and expect a natural language answer in return. To do this, you'll use the textCompletion flow that was created in the first tutorial in this series, and modify the prompt template to match the desired prompt template structure for Llama 3.1, as shown in their documentation.

The following code is what a text completion flow in watsonx flows engine looks like.

flows="""
    textCompletion = templatedPrompt(promptTemplate: "{question}") | completion(model: textCompletion.model, parameters: textCompletion.parameters)
"""

In the previous flow example, you can use the templatedPrompt step to define a prompt template, in which you can subsitute a variable (in this case {question}) with the natural language question when you send your request.

To add the prompt template, you can copy-paste the following code into your wxflows.toml file.

flows="""
    textCompletion = templatedPrompt(promptTemplate: "<|begin_of_text|><|start_header_id|>system<|end_header_id|>You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>{question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>") | completion(model: textCompletion.model, parameters: textCompletion.parameters)
"""

The prompt template is setting the system prompt to tell the LLM to take the role of a helpful assistant, and sets the user prompt to the output of the {question} variable.

To try out this flow, you must first deploy the flow to your flows engine endpoint by running the following command in your terminal.

wxflows deploy

You can see the endpoint that the flow is deployed to printed in your terminal. To interact with that endpoint, you can use the SDK for JavaScript or Python. The first tutorial in this series showed how to set up watsonx flows engine and how to use the JavaScript SDK. You can use the updated textCompletion flow with the example in that previous tutorial by modifying the following JavaScript snippet in the app/index.js file.

const flowName = "textCompletion"
const question = "Who is the president of France?"

const result = await model.flow({
    schema,
    flowName,
    variables: {
        question,
        model: "meta-llama/llama-3-1-8b-instruct",
        parameters: {
            stop_sequences: []
        },
    },
})

To run this file, you can execute node index.js from your command line. The question "Who is the president of France?" is injected into the base prompt, and the answer is printed in your terminal.

With the textCompletion flow done, let's put some perspective to the large context window offered by the Llama 3.1 model.

Large context window

One of the advantages of Meta's Llama 3.1 is the large context window, which is helpful when using this model with large documents or a long chat history. This is taking away a lot of complexity because you no longer need the RAG pattern for small to midsized data sets. The Llama 3.1 model has a context window length of 131,072 (input + output), which is limited to 4,096 tokens for the 8b version and 16,384 for the 405b model when using watsonx.ai as the LLM provider for flows engine.

To put this large context window into some perspective: on average 1 token corresponds to 4 characters of text, and the average common English word is 5 characters long. Following this rule of thumb with Llama 3.1, the maximum context window is 524,288 characters or 104,858 words of common English. For reference, a general guideline is that one page in a book is 500 words (single spaced), meaning the context window for Llama 3.1 is 209 pages. This means you could insert the entire contents of the book "The Great Gatsby" or the first Harry Potter novel.

For performance reasons, you cannot use the complete large context window when using watsonx flows engine on a free plan, but the 16,384 token context window still gives you the option to insert a large file, such as a company's annual statement or a multiturn chat history.

The prompt template that you set up for text completion also works for a multiturn conversation because you can iterate the "user" and "assistant" sections.

<|begin_of_text|>
<|start_header_id|>system<|end_header_id|>
    {SYSTEM PROMPT}
<|start_header_id|>user<|end_header_id|>
    {FIRST USER MESSAGE}
<|start_header_id|>assistant<|end_header_id|>
    {FIRST MODEL RESPONSE}
<|start_header_id|>user<|end_header_id|>
    {FOLLOW UP USER MESSAGE}
<|start_header_id|>assistant<|end_header_id|>
    {FOLLOW UP MODEL RESPONSE}

// iterate dozes of times

Because Llama 3.1 has a very large context window, it is able to reason across a larger chat history than most other models. For example, you can use this multiturn chat to summarize multiple blog posts and ask follow-up questions.

The next section describes using Meta Llama 3.1 for code to natural language.

Code to natural language

The examples in this tutorial end with exploring the potential of Meta's Llama 3.1 in transforming code into natural language, in this case SQL to natural language. Llama 3.1 is trained on large data sets, including code, and is able to handle various programming languages. In general, most LLMs are able to work with programming languages that have a declarative design, such as SQL.

Assume that you have the following database schema:

CREATE TABLE countries (
    country_id CHAR (2) PRIMARY KEY,
    country_name VARCHAR (40) DEFAULT NULL,
);

CREATE TABLE locations (
    location_id INT (11) AUTO_INCREMENT PRIMARY KEY,
    street_address VARCHAR (40) DEFAULT NULL,
    postal_code VARCHAR (12) DEFAULT NULL,
    city VARCHAR (30) NOT NULL,
    state_province VARCHAR (25) DEFAULT NULL,
    country_id CHAR (2) NOT NULL,
    FOREIGN KEY (country_id) REFERENCES countries (country_id) ON DELETE CASCADE ON UPDATE CASCADE
);

CREATE TABLE departments (
    department_id INT (11) AUTO_INCREMENT PRIMARY KEY,
    department_name VARCHAR (30) NOT NULL,
    location_id INT (11) DEFAULT NULL,
    FOREIGN KEY (location_id) REFERENCES locations (location_id) ON DELETE CASCADE ON UPDATE CASCADE
);

CREATE TABLE employees (
    employee_id INT (11) AUTO_INCREMENT PRIMARY KEY,
    first_name VARCHAR (20) DEFAULT NULL,
    last_name VARCHAR (25) NOT NULL,
    email VARCHAR (100) NOT NULL,
    phone_number VARCHAR (20) DEFAULT NULL,
    hire_date DATE NOT NULL,
    role VARCHAR (25) NOT NULL,
    salary DECIMAL (8, 2) NOT NULL,
    manager_id INT (11) DEFAULT NULL,
    department_id INT (11) DEFAULT NULL,
    FOREIGN KEY (department_id) REFERENCES departments (department_id) ON DELETE CASCADE ON UPDATE CASCADE,
    FOREIGN KEY (manager_id) REFERENCES employees (employee_id)
);
`

In this database schema, you can find four tables, and some of these tables are linked to other tables using foreign keys. For example, there's a table called employees that references itself and the departments table. Departments are linked to an office location, and these locations are linked to a country.

For a developer or data scienists who is confident in writing SQL queries, it should be straightforward to retrieve the data they want. But, let's assume that you're a junior developer learning SQL or a non-technical analyst that needs to retrieve data for an analysis. Then, you can use Llama 3.1 to generate an SQL query, and you only need to copy-paste into a system with access to the database.

Add the following flows to your wxflows.toml file:

flows=""""
    codeToNL = templatedPrompt(promptTemplate: "<|begin_of_text|><|start_header_id|>system<|end_header_id|>You are a helpful assistant<|eot_id|><|start_header_id|>user<|end_header_id|>Based on the table schema, question, SQL query, and SQL response below, write a new SQL response. Be concise, just output the SQL response. {question}<|eot_id|><|start_header_id|>assistant<|end_header_id|>") | completion(model: "meta-llama/llama-3-1-8b-instruct", parameters: codeToNL.parameters)
""""

The prompt template now contains instructions to generate an SQL query that is based on a given question and schema. Also, the meta-llama/llama-3-1-8b-instruct model has been set as the default model to use.

Before you can use this flow, it must be deployed to your flows engine endpoint by running the following command.

wxflows deploy

To get the SQL query for a natural language question by using the new codeToNL flow, you must make some small changes to the app/index.js file:

// Make sure these match your values in `wxflows.toml'
const flowName = 'codeToNL'
const question = `
    Question: give me a list of employees and the countries they work from.

    Schema: """
        CREATE TABLE countries (
            country_id CHAR (2) PRIMARY KEY,
            country_name VARCHAR (40) DEFAULT NULL,
        );

        CREATE TABLE locations (
            location_id INT (11) AUTO_INCREMENT PRIMARY KEY,
            street_address VARCHAR (40) DEFAULT NULL,
            postal_code VARCHAR (12) DEFAULT NULL,
            city VARCHAR (30) NOT NULL,
            state_province VARCHAR (25) DEFAULT NULL,
            country_id CHAR (2) NOT NULL,
            FOREIGN KEY (country_id) REFERENCES countries (country_id) ON DELETE CASCADE ON UPDATE CASCADE
        );

        CREATE TABLE departments (
            department_id INT (11) AUTO_INCREMENT PRIMARY KEY,
            department_name VARCHAR (30) NOT NULL,
            location_id INT (11) DEFAULT NULL,
            FOREIGN KEY (location_id) REFERENCES locations (location_id) ON DELETE CASCADE ON UPDATE CASCADE
        );

        CREATE TABLE employees (
            employee_id INT (11) AUTO_INCREMENT PRIMARY KEY,
            first_name VARCHAR (20) DEFAULT NULL,
            last_name VARCHAR (25) NOT NULL,
            email VARCHAR (100) NOT NULL,
            phone_number VARCHAR (20) DEFAULT NULL,
            hire_date DATE NOT NULL,
            role VARCHAR (25) NOT NULL,
            salary DECIMAL (8, 2) NOT NULL,
            manager_id INT (11) DEFAULT NULL,
            department_id INT (11) DEFAULT NULL,
            FOREIGN KEY (department_id) REFERENCES departments (department_id) ON DELETE CASCADE ON UPDATE CASCADE,
            FOREIGN KEY (manager_id) REFERENCES employees (employee_id)
        );
    """
    `
    const result = await model.flow({
        schema,
        flowName,
        variables: {
            question,
            parameters: {
                stop_sequences: []
            },
        },
    })

The name of the flow used in the previous code block is codeToNL, and the value for question contains the complete schema for the database and the question "give me a list of employees and the countries they work from."

To get the response from the LLM, you must run the JavaScript code:

node index.js

In your terminal, the following SQL query is (most likely) printed:

SELECT e.first_name, e.last_name, l.country_id, c.country_name FROM employees e INNER JOIN locations l ON e.department_id = l.location_id INNER JOIN countries c ON l.country_id = c.country_id

The logical next step is to take this SQL query and use it in a system that has access to the database, or take the previous JavaScript code and integrate it with a library to connect directly to the SQL database. Llama 3.1 also supports tool calling, meaning that you can turn the database retrieval into a tool and ask follow-up questions or combine the database results with knowledge from the LLM or other tools.

The next tutorial covers how to use tool calling with watsonx flows engine, so stay tuned!

What's next?

In this tutorial, you learned how to use Meta Llama 3.1 LLMs within watsonx.ai flows engine. You learned about the key capabilities of Llama 3.1, including handling large context windows and understanding code generation, by setting up tailored AI flows. The flexibility of watsonx flows engine lets you experiment with different models so that you can deploy AI applications effectively.

The final tutorial in this series will cover using watsonx.ai flows engine with Mistral, and if you have any questions or feedback make sure to let us know on Discord!