Foundational models, a form of generative artificial (Gen AI) intelligence, are machine learning models that are trained on broad data such that it can be applied across a wide range of use cases. They’re making it easier access to Gen AI models.

LLaVA, or Large Language and Vision Assistant, is a Llama2-based multimodal large language model (LLM) that uses the CLIP visual encoder with Vicuna, resulting in outstanding performance in visual chats.

LLaVA can be used for similar product searches (ecommerce), step-by-step guides on equipment repair (manufacturing ), highly accurate OCR for Data capture, and other industrial use cases.

In this tutorial, take a look at how to run LLaVa on IBM Cloud Virtual Servers.

Prerequisites

To follow this tutorial, you need a basic understanding of IBM Cloud.

Estimated time

It should take you approximately 10 - 15 minutes to complete the tutorial.

Steps

Step 1. Set up the environment

This tutorial uses IBM Cloud. You will provision and use the following components.

Ubuntu 22.04 - 16 vCPU | 80 GiB | 1 x NVIDIA L4 24 GB Virtual server instances
- Provisioning a VSI auto creates the following for you:
  - A VPC attached to it
  - A security group governing the previous VPC
A Floating IP in the same region to expose your app to the internet

Provision a Virtual server for VPC.
Name the instance.
- For image options, pick ibm-ubuntu-22-04-4-minimal-amd64-1
- For Profile, select a 16 vCPU | 80 GiB | 1 x NVIDIA L4 24 GB
  
  Although LlaVa 7b can run in 8 GB of RAM and CPU only, it’s extremely slow. Therefore, you should use the previous configuration after numerous test runs.
Generate an SSH Key that is specific to the system that you will use to log in. You must click Create SSH key, then name the key. Click Create. The key is auto generated and downloaded for you.
Choose Virtual network interface for Networking, and let it create one for you.

That’s it. Now wait for the provisioning to complete.

Step 2. Set up the networking

Before you can access the cluster, you must also set up the networking.

Step 2a. Get a Floating IP

The instance does not come with a Floating IP, so you must get one.

Navigate to Floating IP under Network from the menu side bar.
Click Reserve, and make sure that you select the same zone as the one you chose for the instance, so you can see it listed in the Resource to bind drop down list.
Click Reserve to assign the Floating IP to your instance.

Step 2b. Allow inbound on Port 80

The app is deployed on port 80 so that it can be accessed without any redirection. Although the required fire wall setting on the system is controlled by the script, you must still manually add an inbound rule to the security policy created.

Navigate to security groups for VPC.
Pick the group that is assigned to your instance (you can find it tagged in the instance details), and navigate to Rules.
Add a new TCP inbound rule to allow traffic on port 80.

That’s it. You can now move to the application side.

Step 3. Access the cluster

Open the terminal, and navigate to the folder where you downloaded the SSH key.
Update the permissions on the key to allow SSH connect.
```
 chmod 400 <path to your pem file>
```

SSH into the cluster by using the following command.

 ssh -i <path to your pem file> root@<Floating IP>

Enter Yes when asked to add the key to known hosts.

Step 4. Installing prerequisites

There is a quick script to install all of the necessary libraries and tools for you as well as a Flask application to give you access to the model over a UI or API. Let’s see how to get it.

Get the required scripts.

 git clone https://github.com/krishnac7/projectHeimdall

cd into the script.
```
 cd projectHeimdall
```
Set up file permissions.
```
 chmod +x install_run_cloud.sh
```
Run the setup files.
```
 ./install_run_cloud.sh
```
Wait for the setup process to complete. It can take a few minutes. After you see the following screen, you are all set.

Note: If you are asked for manual input, especially during the update-initramfs step, press enter until you see the console again. The process will then continue.

If you are deploying on an instance with GPU, you must manually reboot the instance after the first run. The app will automatically re-run.

Step 5: Access the UI

Open the URL: http://<Floatingip > to access the UI.

Gif showing app

Conclusion

You successfully deployed a multimodal LLM for a visual chat use case on IBM Cloud, which makes it easy to obtain high-compute GPUs and deploy complex applications on top of it in matter of minutes.

Deploy multimodal LLMs on IBM Cloud Virtual Server with GPU

Deploy multimodal LLMs on IBM Cloud Virtual Server with GPU