Data Prep Kit (DPK)

Introducing Data Prep Kit (DPK) as a streamlined data preparation tool for LLMs

The Data Prep Kit (DPK) offers a diverse set of pre-built transforms to streamline data preparation for AI applications. These transforms cater to various data formats, including text, code, and structured data. These built-in transforms provide a strong foundation for handling common data processing tasks. This rich collection of built-in transforms empowers one to efficiently prepare the data for various AI and machine learning applications.

While Data Prep Kit includes several built-in transforms, you can also build custom data preparation transforms to meet your needs. In this tutorial, learn how to implement a Processing transform that calculates a signature value for a document and stores the signature as part of the metadata associated with the document.

In this tutorial, you learn how to use the Data Prep Kit (DPK) to prepare data for fine-tuning and to achieve the business use case of contract analysis.

Data Prep Kit (DPK) is a scalable, flexible, robust, and easy to use framework for data processing. Data Prep Kit is data agnostic, handling diverse data formats including text, code, and structured data. It uses distributed computing frameworksvlike Ray and Spark to efficiently process large data sets. Developers can createvcustom transforms (to supplement the built-in transforms) to readily address specificvdata processing needs.

In this learning path, learn how to use Data Prep Kit (DPK) to prepare data for large language model (LLM) applications.

In this tutorial, learn how to build a Ray runtime, create unit tests, integrate with CI/CD, and add KFP extensions to existing transforms.

In this tutorial, you learn how to use the Data Prep Kit (DPK) to prepare PDFs for a RAG pipeline.