Deploying LLMs tuned with Parameter-Efficient Fine-Tuning (PEFT) techniques in watsonx.ai

IBM watsonx.ai can deploy models that have been fine-tuned with different methods. One such method is called Parameter-Efficient Fine-Tuning (PEFT), which is a powerful technique that enables the efficient adaptation of pre-trained models to specific tasks, without requiring significant computational resources or large amounts of training data. By using PEFT, you can adapt the model to your specific needs, while minimizing the risk of overfitting.

For example, consider a scenario where you want to use a pre-trained language model to classify text into different categories, but the pre-trained model is not specifically designed for your particular classification task. By using PEFT, you can fine-tune the model to adapt to your specific classification task, without requiring a large amount of labeled training data.

Another example is in the field of natural language processing, where PEFT can be used to adapt a pre-trained language model to a specific language or dialect, without requiring a large amount of training data.

In this tutorial, we will walk you through the process of deploying models with PEFT techniques in watsonx.ai by using the REST API. We will use LoRA and QLoRA adapters. These adapters are designed to be used with the watsonx.ai platform and can be easily integrated into your existing workflows.

Types of PEFT techniques

Let’s compare these common PEFT techniques: Low-Rank Adaptation of LLMs (LoRA) and Quantized Low-Rank Adaptation (QLoRA).

LoRA is a parameter-efficient fine-tuning technique that adapts a pre-trained language model to a specific task or data set. It achieves this by updating a low-rank approximation of the model's weight matrix, which provides efficient adaptation without requiring significant updates to the entire model.
QLoRA is a variant of LoRA that incorporates quantization to further reduce the memory footprint and computational resources required. By quantizing the weights of the model, QLoRA reduces the precision of the weights from 32-bit floating-point numbers to lower precision, such as 4-bit or 8-bit.

Metric	LoRA	QLoRA
Weight updates	Updates a low-rank approximation of the model's weight matrix.	Updates a low-rank approximation of the model's weight matrix with quantization.
Quantization	No quantization.	Quantizes the weights to lower precision such as 4-bit or 8-bit.
Memory footprint	Smaller memory footprint compared to traditional fine-tuning.	Even smaller memory footprint due to quantization.
Computational resources	Requires less computational resources compared to traditional fine-tuning.	Requires even less computational resources due to quantization.
Performance	Maintains performance comparable to traditional fine-tuning.	Maintains performance comparable to LoRA, with some potential degradation due to quantization.
Training time	Faster training time compared to traditional fine-tuning.	Faster training time compared to LoRA, due to quantization.

Prerequisites

You must have the following accounts:
- IBM Cloud account
- Watsonx.ai trial account
You must fine-tune a foundation model and train it by using the REST API. For more information, see Tuning a foundation model programmatically in the watsonx.ai documentation.
You must add the IBM shipped foundation model to watsonx.ai. For more information, see Adding foundation models to IBM watsonx.ai in the IBM Software Hub documentation.

Steps

For fine-tuning a model using PEFT techniques and then deploying that model on watsonx.ai, complete these steps:

Review the requirements
Optional: Create a custom hardware specification
Create a repository asset for the base foundation model
Deploy the base foundation model
Create a repository asset for the LoRA / QLoRA adapter
Deploy the LoRA or QLoRA adapter
Inference your deployment

Step 1. Review the requirements

The watsonx.ai cluster on which you want to deploy your fine-tuned model must meet the following requirements:

Configure RedHat OpenShift AI on IBM Software Hub.
The cluster should not be configured with NVIDIA MIG. Fine tuning is not compatible with MIG in the watsonx 2.1.1 release.
Your fine-tuned model must meet these requirements:
- It must be built with an architecture that is supported for deployment on watsonx.ai.
- It must be deployed on supported hardware.
- It must be deployed with the supported software specification.

To learn more about the requirements, see Requirements for deploying PEFT models in the watsonx.ai documentation.

Step 2. (Optional) Create a custom hardware specification

You might need to create a customized hardware specification if your model requires a specific configuration which is not met by predefined specs, its size doesn't fit predefined ranges, or you want to optimize performance or cost-effectiveness.

To learn more about calculating the GPU memory required for LoRA or QLoRA models, see Calculating GPU memory for LoRA and QLoRA models in the watsonx.ai documentation.

The following code sample shows how to create a custom hardware specification with REST API:

curl -ik -X POST -H "Authorization: Bearer $TOKEN" "https://<HOST>/v2/hardware_specifications?space_id=$space_id" \
-H "Content-Type:application/json" \
--data '{
  "name": "custom_hw_spec",
  "description": "Custom hardware specification for foundation models",
  "nodes": {
    "cpu": {
      "units": "2"
    },
    "mem": {
      "size": "128Gi"
    },
    "gpu": {
      "num_gpu": 1
    }
  }
}'

Step 3. Create a repository asset for the base foundation model

The following code sample shows how to create the base foundation model asset for a supported foundation model:

curl -X POST "https://<HOST>/ml/v4/models?version=2024-01-29" \
-H "Authorization: Bearer <token>" \
-H "content-type: application/json" \
--data '{
  "name":"base foundation model asset",
  "space_id":"<space_id>",
  "foundation_model":{
    "model_id":"ibm/granite-3-1-8b-base"
  },
  "type":"base_foundation_model_1.0",
  "software_spec":{
    "name":"watsonx-cfm-caikit-1.1"
  }
}'

Step 4. Deploy the base foundation model

When you create an online deployment for your base foundation model, you must set the enable_lora parameter to true in the JSON payload so that you can deploy the LoRA or QLoRA adapters by using the base foundation model.

The following code sample shows how to create an online deployment for the base foundation model asset with REST API:

curl -X POST "https://<HOST>/ml/v4/deployments?version=2024-01-29" \
-H "Authorization: Bearer <token>" \
-H "content-type: application/json" \
--data '{
  "asset":{
    "id": "<asset_id>"  // WML base foundation model asset
  },
  "online":{
    "parameters":{
       "serving_name": "basefmservingname",
       "foundation_model": {
              "max_batch_weight": 10000,
              "max_sequence_length": 8192,
              "enable_lora": true,
              "max_gpu_loras": 8,                                
              "max_cpu_loras": 16,            
              "max_lora_rank": 32
       }
    }
  },
  "hardware_spec": {                        // Only one, of "id" or "name" must be set. 
    "id": "<hardware_spec_id>",
    "num_nodes": 1
  },
  "description": "Testing deployment using base foundation model",
  "name":"base_fm_deployment",
  "space_id": "<space_id>"  // Either "project_id" (or) "space_id". Only "one" is allowed
}'

Step 5. Create a repository asset for the LoRA or QLoRA adapters

While training the model, if you did not enable the auto_update_model option, you must create a repository asset for the LoRA or QLoRA adapters.

If the auto_update_model option was enabled during training, the LoRA adapter model asset is already created in the Watson Machine Learning repository. In that case, you can proceed with creating a deployment for the LoRA adapter model asset.

The following code sample shows how to create the LoRA adapter model asset:

curl -X POST "https://<HOST>/ml/v4/models?version=2024-01-29" \
-H "Authorization: Bearer <token>" \
-H "content-type: application/json" \
--data '{
  "name":"lora adapter model asset",
  "space_id":"<space_id>",
  "foundation_model":{
    "model_id":"finetunedadapter-385316ea-1d25-4274-9550-e387a1355241"
  },
  "type":"lora_adapter_1.0",
  "software_spec":{
    "name":"watsonx-cfm-caikit-1.1"
  },
  "training":{
    "base_model":{
      "model_id":"ibm/granite-3-1-8b-base"
    },
    "task_id":"summarization",
    "fine_tuning": {
         "peft_parameters": {
             "type": "lora",
             "rank": 8
         }
    }
  }
}'

Step 6. Deploy the LoRA or QLoRA adapter model asset

Use the deployed base foundation model to deploy the LoRA adapters as an additional layer on the base foundation model.

The following code sample shows how to create a deployment for the LoRA adapter model:

curl -X POST "https://<HOST>/ml/v4/deployments?version=2024-01-29" \
-H "Authorization: Bearer <token>" \
-H "content-type: application/json" \
--data '{
  "asset":{
    "id":<asset_id>  // WML lora adapter model asset
  },
  "online":{
    "parameters":{
      "serving_name":"lora_adapter_dep"
    }
  },
  "base_deployment_id": "<replace with your WML base foundation model deployment ID>",
  "description": "Testing deployment using lora adapter model",
  "name":"lora_adapter_deployment",
  "space_id":<space_id>  // Either "project_id" (or) "space_id". Only "one" is allowed
}'

Step 7. Inference your deployment

After deploying the PEFT model, you can inference the model by providing text or stream input data to the deployed model to generate predictions in real-time.

To generate a text response from your deployed PEFT model, use the following code sample:

curl -X POST "https://<HOST>/ml/v1/deployments/<deployment_id>/text/generation?version=2024-01-29" \
-H "Authorization: Bearer <token>" \
-H "content-type: application/json" \
--data '{
 "input": "What is the boiling point of water?",
 "parameters": {
    "max_new_tokens": 200,
    "min_new_tokens": 20
 }
}'

To generate a stream response from your deployed PEFT model, use the following code sample:

curl -X POST "https://<HOST>/ml/v1/deployments/<deployment_id>/text/generation_stream?version=2024-01-29" \
-H "Authorization: Bearer <token>" \
-H "content-type: application/json" \
--data '{
 "input": "What is the boiling point of water?",
 "parameters": {
    "max_new_tokens": 200,
    "min_new_tokens": 20
 }
}'

Congratulations, your model is now fully deployed and ready to use!

Summary and next steps

Deploying models fine tuned with PEFT techniques on watsonx.ai is a powerful way to adapt pre-trained models to specific tasks and improve their performance. By following the steps outlined in this tutorial, you can deploy models with PEFT techniques and start taking advantage of the benefits of this powerful technique.

Get started with a free trial of watsonx.ai today!

Learn more in the watsonx.ai documentation:

If you encounter any issues during the deployment process, you can see troubleshooting tips for deploying PEFT models.
To learn about other model customization options, see Methods for tuning foundation models.
To learn more about deploying tuned models, see Deploying tuned models.

Products

Languages

Technologies

All Events

External Resources