IBM Developer

Tutorial

Deploy multimodal LLMs on IBM Cloud Virtual Server with GPU

Learn how to run LLaVa on IBM Cloud Virtual Servers to obtain high-compute GPUs and deploy complex applications on top of it

By Krishna Balaga, Rachana Vishwanathula

On this page

Foundational models, a form of generative artificial (Gen AI) intelligence, are machine learning models that are trained on broad data such that it can be applied across a wide range of use cases. They’re making it easier access to Gen AI models.

LLaVA, or Large Language and Vision Assistant, is a Llama2-based multimodal large language model (LLM) that uses the CLIP visual encoder with Vicuna, resulting in outstanding performance in visual chats.

LLaVA can be used for similar product searches (ecommerce), step-by-step guides on equipment repair (manufacturing ), highly accurate OCR for Data capture, and other industrial use cases.

In this tutorial, take a look at how to run LLaVa on IBM Cloud Virtual Servers.

Prerequisites

To follow this tutorial, you need a basic understanding of IBM Cloud.

Estimated time

It should take you approximately 10 - 15 minutes to complete the tutorial.

Steps

Step 1. Set up the environment

This tutorial uses IBM Cloud. You will provision and use the following components.

  • Ubuntu 22.04 - 16 vCPU | 80 GiB | 1 x NVIDIA L4 24 GB Virtual server instances
    • Provisioning a VSI auto creates the following for you:
      • A VPC attached to it
      • A security group governing the previous VPC
  • A Floating IP in the same region to expose your app to the internet

  1. Provision a Virtual server for VPC.

  2. Name the instance.

    • For image options, pick ibm-ubuntu-22-04-4-minimal-amd64-1

    • For Profile, select a 16 vCPU | 80 GiB | 1 x NVIDIA L4 24 GB

      Image and profile

      Although LlaVa 7b can run in 8 GB of RAM and CPU only, it’s extremely slow. Therefore, you should use the previous configuration after numerous test runs.

  3. Generate an SSH Key that is specific to the system that you will use to log in. You must click Create SSH key, then name the key. Click Create. The key is auto generated and downloaded for you.

    Creating the SSH key

  4. Choose Virtual network interface for Networking, and let it create one for you.

    Networking

That’s it. Now wait for the provisioning to complete.

Step 2. Set up the networking

Before you can access the cluster, you must also set up the networking.

Step 2a. Get a Floating IP

The instance does not come with a Floating IP, so you must get one.

  1. Navigate to Floating IP under Network from the menu side bar.

    Floating IP in menu

  2. Click Reserve, and make sure that you select the same zone as the one you chose for the instance, so you can see it listed in the Resource to bind drop down list.

    Resource to bind

  3. Click Reserve to assign the Floating IP to your instance.

Step 2b. Allow inbound on Port 80

The app is deployed on port 80 so that it can be accessed without any redirection. Although the required fire wall setting on the system is controlled by the script, you must still manually add an inbound rule to the security policy created.

  1. Navigate to security groups for VPC.

  2. Pick the group that is assigned to your instance (you can find it tagged in the instance details), and navigate to Rules.

  3. Add a new TCP inbound rule to allow traffic on port 80.

    TCP Inbound rule

That’s it. You can now move to the application side.

Step 3. Access the cluster

  1. Open the terminal, and navigate to the folder where you downloaded the SSH key.

  2. Update the permissions on the key to allow SSH connect.

     chmod 400 <path to your pem file>

  3. SSH into the cluster by using the following command.

     ssh -i <path to your pem file> root@<Floating IP>

  4. Enter Yes when asked to add the key to known hosts.

    Adding key

Step 4. Installing prerequisites

There is a quick script to install all of the necessary libraries and tools for you as well as a Flask application to give you access to the model over a UI or API. Let’s see how to get it.

  1. Get the required scripts.

     git clone https://github.com/krishnac7/projectHeimdall

  2. cd into the script.

     cd projectHeimdall

  3. Set up file permissions.

     chmod +x install_run_cloud.sh

  4. Run the setup files.

     ./install_run_cloud.sh

    Running setup files

  5. Wait for the setup process to complete. It can take a few minutes. After you see the following screen, you are all set.

    Completion screen

    Note: If you are asked for manual input, especially during the update-initramfs step, press enter until you see the console again. The process will then continue.

    If you are deploying on an instance with GPU, you must manually reboot the instance after the first run. The app will automatically re-run.

Step 5: Access the UI

Open the URL: http://<Floatingip > to access the UI.

Gif showing app

Conclusion

You successfully deployed a multimodal LLM for a visual chat use case on IBM Cloud, which makes it easy to obtain high-compute GPUs and deploy complex applications on top of it in matter of minutes.