Hugging Face and IBM working together in open source

Open source brings together communities and companies to accomplish common goals. By sharing expertise and investments, state-of-the-art technologies can be advanced with open standards and transparency. This process fosters safe, ethical innovation and reduces cost. Hugging Face is using open source to accomplish their mission to democratize good machine learning and build the AI community of the future.

This blog is part of a series that focuses on the key open source projects that underpin the watsonx platform to scale and accelerate AI. With watsonx, you can use open source frameworks and tools for code-based, automated, and visual data science capabilities - all in a secure, trusted studio environment. Watsonx lets you quickly tune models for your specific business needs using the latest open source and IBM-trained foundation models. This blog explains how Hugging Face and IBM are working together in the open source space to advance state-of-the-art enterprise AI.

The Hugging Face AI community

Hugging Face built an incredibly popular AI community around open source libraries, models, and data sets. The open source transformers library has over 100,000 GitHub stars and has been a unifying force in the AI community. The Hugging Face hub now hosts more than 230,000 models and over 40,000 data sets, and those numbers are growing quickly.

One ubiquitous open source project or one omnipresent open source hub alone is impressive enough, but Hugging Face combines many projects and the hub, and, perhaps most importantly, has built a community around AI.

Having a common denominator helps bring a community together in a way that creates a multiplier effect. Hugging Face is creating de facto standards that help to rapidly advance the state-of-the-art in AI.

IBM open source leadership

IBM has decades of leadership in the open source space. Going back to the early days of Linux® and the web, IBM has been a strong proponent of open standards, open governance, and open source practices to create transparent interaction across communities and with IBM customers. In 2022, IBM and Red Hat were two of the top six open source contributors in the world (when counted separately).

IBM also brings decades of experience in groundbreaking AI. Now, IBM's next generation platform -- watsonx.ai -- is using an open source foundation model stack to help businesses create value with generative AI.

IBM and Hugging Face working together in open source

To be a good open source community citizen, you need to give back. IBM has been contributing to a variety of Hugging Face projects, including the Text Generation Inference server (TGIS) and the Parameter-Efficient Fine-Tuning (peft) projects, both of which are key parts of watsonx. IBM continues to give back by contributing algorithms and implementations to these projects and by creating new projects.

IBM created a new open source project called Caikit. Caikit is an AI toolkit that lets users serve, train, and deploy models through a set of developer-friendly APIs. With Caikit, Hugging Face models can be containerized and served on any cloud (or on prem) with consistent interfaces for the applications that rely on the models for predictions. Where the Hugging Face transformers library brings a unifying API for using models in Python code, Caikit adds cloud-native layers with consistent industry-standard APIs.

Using TGIS, peft, and Caikit, watsonx allows large pretrained foundation models to be tuned with a small amount of business domain data to create new capabilities for businesses. Hugging Face and IBM technology helps customers innovate faster and build more complex solutions. Open source is essential in selecting the best models, training, optimizing, and running at scale.

Watsonx plus Hugging Face for the enterprise

Watsonx is designed for the enterprise and is targeted for business domains, empowering value creators to transform business applications into AI-first applications. Using watsonx, you can use general-purpose, pretrained foundation models and fine-tune them for your business needs. Choose from the rapidly growing set of models that are available from Hugging Face, and use prompt-tuning to guide their responses. IBM also provides a suite of curated foundation models that are trained to ensure model trust and efficiency in business applications. Whichever model you start with, you have the tools to quickly tune a model for your specific business needs.

Building AI into your business is not without risks, but by leveraging an evaluation criteria built on principles for trust and transparency you can reduce the risk to your enterprise and reap the benefits of AI technologies. Watsonx can help you avoid choosing a model that contains biased or untrustworthy data, or one that was not created under the umbrella of an ethics-focused governance model.

Bring it home

With open source, Hugging Face, and watsonx, IBM brings the rapidly evolving world of AI - tuned to your business needs - to your enterprise hybrid cloud. IBM is committed to supporting your hybrid cloud or multicloud environments. You can run your solution on prem, or you can deploy it to any cloud with Red Hat OpenShift, an enterprise-grade, open source application platform. Together we support the entire AI lifecycle with tools, security, governance, and services for all enterprise needs.

Watsonx.ai brings together new generative AI capabilities powered by foundation models and traditional machine learning into a powerful platform that spans the AI lifecycle. With watsonx.ai, you can train, validate, tune, and deploy models with ease, and build AI applications in a fraction of the time with a fraction of the data. Try the watsonx.ai next-generation studio for AI builders.