Article

Use generative AI in intelligent workflow automation with the IBM watsonx platform

Automate segments of the talent acquisition process by using watsonx technology

By

Supal Chowdhury,

Anand Bhushan

On this page

In today's dynamic business landscape, organizations across various industries are grappling with many challenges that hamper their efficiency, productivity, and their ability to deliver exceptional customer service. These challenges range from data management and decision-making to workflow automation and regulatory compliance. These are complex issues that require intelligent, scalable, and flexible solutions. This is where generative AI, and more specifically, the IBM watsonx platform, can help. By applying generative AI, organizations are able to improve their efficiency, productivity, decision-making, and customer service.

This article (and the accompanying tutorial) provides a guideline for creating and executing a sample generative AI-driven automation case study within the talent acquisition process. The scenario uses the capabilities of the IBM watsonx platform, including watsonx.ai, watsonx.data, and watsonx Assistant. The article explains the difficulties that are associated with generative AI, and presents a solution that uses IBM watsonx. It uses a fictitious organization to show how a business might intelligently automate segments of its talent acquisition process (profile screening, assessment, and onboarding) by using IBM watsonx technology.

watsonx.ai

Part of the IBM watxonx platform, watsonx.ai brings together generative AI capabilities, powered by foundation models and traditional machine learning into a powerful studio spanning the AI lifecycle. watsonx.ai lets you easily train, validate, tune, and deploy generative AI, foundation models and machine learning capabilities and build AI applications in a fraction of the time with a fraction of the data.

watsonx.data

A lakehouse is a combination of a data lake (for example, unstructured, low-cost, flexibility, and s3 object storage) and a warehouse (for example, ACID, CRUD, and ETL). IBM watsonx.data is an IBM lakehouse, which is an open data lakehouse architecture that is built on Presto. Watsonx.data combines the open, flexible, and low-cost storage aspects of data lakes with the transactional qualities and performance of a data warehouse. It brings together the advantages of data warehouses and data lakes by supporting open tables and file formats and by leveraging open source technologies such as the Presto engine and Apache Spark. It enables organizations to store data on low-cost storage while ensuring that the data is open. The following image shows the watsonx.data lakehouse architecture with integrated layers.

watxonx.data lakehouse architecture

The following figure shows the key components of the IBM lakehouse, multiple query engines, open table formats, and built-in enterprise governance.

Lakehouse

Watsonx.data is built on three primary integrated components:

  • Engines: Engines are open source, fast, reliable, and highly scalable SQL query engines (for example, Presto, an open source query engine).
  • Catalogs: Catalogs track a table's metadata (for example, Apache Hive and Apache Iceberg).
  • Storage: Storage combines Buckets (S3) and relational databases or schemas (for example, PostgreSQL and Db2) that the query engines directly access.

The following image shows the watsonx.data infrastructure management as an enabler for accessing data across the enterprise for internal resource management.

Core components

watsonx Assistant

Watsonx Assistant is an AI-driven virtual agent that is designed to deliver prompt, consistent, and precise responses to customers through various messaging platforms, applications, devices, and channels. With watsonx Assistant, you can integrate conversational interfaces into any application, device, or channel. Unlike most virtual assistants that attempt to emulate human interactions, watsonx Assistant can determine when to retrieve information from a knowledge base, when to seek clarification, and when to connect the user with a human agent.

See the watsonx Assistant architecture for more information.

Business challenges

Today, one of the most pressing challenges that businesses face is managing the enormous amount of data that they generate and collect. If used correctly, this data provides valuable insights that can drive business growth. However, the sheer volume of data often makes it difficult to process and analyze. Watsonx.data provides a fit-for-purpose data store that enables businesses to connect to data in minutes, get trusted insights, and reduce their data warehouse costs.

Many businesses still rely on outdated, manual processes for tasks that can be automated, leading to inefficiencies and errors. Generative AI can help automate these workflows intelligently. For instance, watsonx Orchestrate lets employees delegate common and complex tasks such as creating job descriptions or generating reports, which frees up employees' time so that they can work on more strategic tasks.

As AI becomes more embedded into daily workflows, businesses face the challenges of ensuring regulatory compliance and addressing ethical concerns. Watsonx.governance lets organizations manage, monitor, and direct their AI activities, mitigating risk and managing regulatory requirements. It uses software automation to strengthen the ability to address these issues without excessive costs or the need to switch your data science platform.

Another business challenge is providing consistent, accurate, and timely customer service across various channels. Watsonx Assistant can help overcome this challenge by understanding customers in the right context and providing fast, accurate answers across any application, device, or channel, enhancing the overall customer service experience.

Sample business case

This article uses a fictitious company called FictCo, whose talent acquisition teams currently perform the following processes for talent acquisition:

  • Profile screening
  • Profile assessment
  • Profile onboarding

While performing the profile screening, the team faces the following challenges when filtering curriculum vitae (CV) to identify the CVs for candidates most suitable for their organization.

  • Reviewing a large number of CVs within a limited time frame is challenging for FictCo. Talent acquisition teams must act quickly to avoid losing top candidates to competitors.
  • Balancing the need to screen a high number of CVs against the need to thoroughly evaluate each applicant's qualifications and fit for the role is a constant challenge.
  • Identifying unqualified candidates who do not meet the minimum job requirements can be time-consuming because of the high volume of applications.
  • Overlooking potentially suitable candidates because FictCo's applicant tracking system might have limitations when accurately matching keywords and identifying relevant candidates.
  • Risking unconscious bias that can influence the decision-making process when filtering a large number of CVs. FictCo talent acquisition teams must remain cautious in their efforts to ensure fair and objective evaluations.
  • Comparing and evaluating candidates consistently because CVs can have various formats.
  • Difficulty in accessing a candidates true potential and value because CVs often lack detailed information about a candidate's background, specific achievements, skills, and project outcomes.
  • Encountering repetitive CVs (old CVs or new CVs with the same candidate), making it hard to distinguish between candidates.
  • Managing a high volume of CVs while ensuring data privacy and compliance with relevant regulations.

This article tackles the profile CV screening process and explains a solution to streamline and automate the process at FictCo, making the process more efficient and effective. By using the capabilities of the IBM watsonx platform, FictCo can intelligently automate the initial CV screening process, ensure a fair and unbiased assessment, manage repetitive CVs, and adhere to data privacy regulations.

Business challenge solution

FictCo used watsonx and a concept called retrieval augmented generation (RAG) in their CV screening process solution. RAG is an AI framework for retrieving facts from an external knowledge base to ground large language models (LLMs) on the most accurate, up-to-date information and to give users insight into LLMs' generative process. FictCo’s process at a high level is:

  1. Index the knowledge base: Generate an embedding for each CV and store it in a vector database (MongoDB).
  2. Retrieve the relevant data from the knowledge base: Find the documents most like the user question from the vector database (MongoDB).
  3. Generate a response: The last step is where foundation models play a role. Take the top (for example, three) number of documents that are returned by the vector database and ask the foundation model to determine an answer to the user query (prompt) by using only the text of the top number of documents.

The following image shows the retrieval augmented generation with watsonx.

RAG with watsonx

The solution for the FictCo CV screening process with RAG is as follows:

  1. CV upload: Team members access a designated webpage that offers the CV screening process. They can upload numerous CVs in various formats such as .pdf or .docx. After uploading the CVs, they click the submit button, which feeds the CVs into watsonx.data for further analysis and model fine-tuning.
  2. Knowledge base vector embeddings: The CV screening solution uses the Hugging Face MiniLM model API to process the uploaded CV files. The information from these CVs is first paragraphed and then an embedding is created. This allows the system to create vectors of the context and nuances in CVs, facilitating the identification of relevant experiences, skills, and achievements. Sentence Transformers all-MiniLM-L6-v2 is used to embed both the knowledge base passages and user queries.
  3. Vector storage: The knowledge base vector embeddings are stored in a NoSQL database, where it can be accessed for real-time queries.
  4. Bot integration: A custom bot integrator must be developed. The talent team selects the CV screening queries by choosing the screening assistant using the watsonx Assistant through the bot integrator and submits the query request.
  5. Interaction with watsonx Assistant: Upon submission, a chat page is initiated for interaction. Users can inquire about matching candidates with specific skill sets required for the role.
  6. Query matching: Create query embeddings and then match the embeddings based on keywords that are related to technical skills, programming languages, or certifications. This enables recruiters to identify candidates with the wanted expertise. The watsonx.ai model can infer context from a candidate's CV and previous work experiences, aiding in the identification of transferable skills and potential for growth. The model aggregates relevant information from various sources to create comprehensive candidate profiles, assisting recruiters in making more informed decisions.
  7. Access to relevant CVs: The watsonx.ai model passes this information to the bot, which responds by providing relevant CV files with hyperlinks for the user to access.

Architectural aspects using the IBM watsonx platform

The following image shows the FictCo profile screening solution workflow.

Profile screening process

Architecture details

Users open a sample web page that is implemented by using the Python Django framework. This page comprises four features that the user can perform.

  1. Uploading sample CVs (for example, five CVs) in a .docx format through the "Feeding CVs" process in watsonx.data using a data ingestion process of loading data. Note: FictCo chose watsonX.data for CV data ingestion because of the following reasons:
    1. It's open and its ability to auto-discover the schema based on the CV file being ingested, which can significantly streamline the data ingestion process by reducing the need for manual schema definition.
    2. It allows CV data integration in a CSV format from any source using SDK/API/CLI/WebConsole, whether on-premises or cloud-based, ensuring flexibility in the volume of CV data handling.
    3. Because it's built on an open lakehouse architecture, which facilitates querying, governance, and sharing of CVs (in a CSV format). This can help FictCo to gain trusted data insights and reduce data management costs by up to 50%.
    4. With built-in governance, security, and automation, it promises a quick start and connection to CV data in various locations.
    5. After the CV data in a CSV format is ingested, watsonX.data serves as a unified data management solution that allows for querying and analyzing all in a single platform.
  2. Creating a knowledge base vector embedding using the Hugging Face MiniLM model and storing it in the vector database (MongoDB) through the CV processor.
  3. Connecting the watsonx Assistant through the bot integrator to open the chat interface. Note: FictCo chose watsonX Assistant as their chatbot interface for the following reasons:
    1. Leveraging deep learning, natural language processing, and machine learning, it offers a chatbot service to streamline CV screening, thereby reducing the time expended by the talent acquisition team.
    2. They wanted to enhance the conversational assistant's performance in the CV screening process by harnessing dialog skills.
    3. It would assist in embedding the chatbot assistant into FictCo's existing webpage.
  4. Doing a prompting base query by using the watsonx.ai API for CV screening. Note: FictCo chose watsonX.ai for getting responses of CV screening queries because of the following reasons:
    1. It offers the Prompt Lab, allowing for experimentation and the creation of precise CV screening prompts.
    2. It excels at extracting essential information from CV screening queries using IBM-trained foundation models.
    3. It supplies comprehensive SDK and API libraries for generating query responses using the foundation model.
    4. It prioritizes security, ensuring the protection of CV data and resources.

Technical design workflow of CV screening process

The entire design comprises four functional areas.

  1. UI/assistant bot: A sample Django-based UI that facilitates CV feeding, embedding, storing, and chatting through the bot integrator of watsonx Assistant.
  2. Uploader: Extracting text from the CV .docx file by using the Python library, creating a CSV file by using the NumPy library, and feeding the file to watsonx.data through an API.
  3. Knowledge base builder: Creating embedded vectors of the CSV files by using the Hugging Face MiniLM model.
  4. Query processor: Submitting the query in the form of a prompt in the chat window to get the response from the best matching embedded vectors of MongoDB.

The following image shows how FictCo created the detail design flow that adheres to the RAG AI framework.

Detail design

Solution achievements

Implementing the IBM watsonx.ai solution into FictCo's recruitment process brings significant value in terms of efficiency, accuracy, fairness, scalability, and continuous improvement. These enhancements lead to a more effective and streamlined recruitment process and include:

  • Increasing the efficiency: With the automation of the initial CV screening, FictCo's recruitment team saves time and can focus on in-depth interviews and assessments. This leads to a more efficient recruitment process where resources are allocated to the most critical tasks.
  • Ensuring scalability: The solution’s ability to handle large volumes of CVs without compromising quality ensures that FictCo can scale up its recruitment efforts as needed, maintaining a consistent and thorough review process regardless of the number of applications received.
  • Eliminating biases: The use of watsonx.ai models helps eliminate human biases and emotions from the evaluation process. Candidates are assessed solely on their qualifications and role suitability, leading to more objective and fair hiring decisions.
  • Improving accuracy: FictCo's solution improves the accuracy by extracting and analyzing relevant information from CVs. This reduces the chances of overlooking qualified candidates or mistakenly selecting unsuitable ones.
  • Efficiently matching candidates: The solution can efficiently match a candidate’s skills with specific job requirements. This allows FictCo to identify the most suitable candidates for each role, improving the overall quality of hires.
  • Sending faster responses: Automating the initial CV screening process ensures that candidates receive faster responses regarding their application status. The personalized feedback also helps candidates understand areas of improvement and indicates that their applications have been adequately considered.
  • Enforcing consistent evaluations: The solution enforces a consistent evaluation process across all CVs, eliminating inconsistencies that might arise due to human biases or varying judgment criteria. This ensures that all candidates’ CVs are evaluated on the same set of standards.
  • Optimizing resources: The solution confirms resource optimization by reducing the time and effort that is spent on manual CV screening. This allows FictCo's recruitment team to allocate their resources more strategically, which might lead to cost savings and improved productivity.
  • Inferring context: The ability of watsonx.ai to infer context from CVs, such as identifying candidates with transferable skills or experiences, even if not explicitly mentioned, allows FictCo to identify potential talent that might otherwise be overlooked.
  • Learning from repetition: The system's ability to learn from new CVs and candidate profiles allows it to continuously improve its screening capabilities and adapt to changing recruitment trends. This ensures that FictCo's recruitment process remains effective and relevant over time.

Conclusion

The rise of generative AI and its integration into contemporary recruitment processes offers opportunities to automate, optimize, and improve hiring practices. Generative AI can help with several challenges in the recruitment industry such as manual CV screening, bias in selection, and inefficiencies due to high volumes of applications. This article examined these business challenges by explaining a sample business case, a CV processing workflow that can be improved by intelligently automating the process by using generative AI and watsonx.

The solution for the FictCo CV processing system involves using the IBM watsonx platform, Python, and other open source technologies to build an intelligent workflow automation system. The development process encompasses several key steps, from architecture and design to implementation:

  • Setting up an environment and preparing the watsonx platform
  • Creating a CV upload UI
  • Storing data by using watsonx.data for analysis
  • Training the AI model by using watsonx.ai
  • Integrating a chatbot by using watsonx Assistant
  • Implementing candidate matching by correct prompting
  • Deploying the solution

Combining generative AI and the IBM watsonx platform presents a transformative approach to talent acquisition. It offers a more efficient, accurate, fair, and scalable solution that not only optimizes the recruitment process for the talent acquisition team, but also improves the candidate experience, making it a valuable asset for any organization.

To experiment with creating the solution, take a look at the accompanying tutorial. Try IBM watsonx.ai for free.