Automate real-time streaming with IBM Bob, Confluent Cloud, and Terraform

Building and deploying real-time streaming systems can be complicated. These systems require complex distributed platforms like Apache Kafka and Flink. Confluent Cloud simplifies this with fully managed services including connectors, Schema Registry, and Flink stream processing. Terraform implements your infrastructure-as-code. IBM Bob makes it even simpler by using spec-driven development (SSD) to generate configurations and code from the ground up.

In this tutorial, you learn how to use IBM Bob to build a banking fraud detection system using Confluent Cloud (Kafka and Flink) and Terraform. You also test the real-time streaming system with simulated banking transactions (produced also by Bob) on Kafka and check the fraud detection system on Confluent Cloud.

Architecture of the fraud detection system

A fraud detection system requires real-time data capabilities. Confluent Cloud provides this real-time data streaming with Apache Kafka and processes (and aggregates) the events in real time with Apache Flink. Although you can manage these resources using Confluent Cloud, the Confluent Terraform provider enables you to define this infrastructure-as-code.

You can build this real-time streaming system by using an IBM Bob skill that generates the necessary configurations and code.

Prerequisites

• IBM Bob installed. Sign up for free trial for Bob.

• HashiCorp Terraform installed. Download and install the Terraform version you need.

• A Confluent Cloud account. Sign up for a free Confluent Cloud account.

Step 1. Create an API key for Confluent Cloud

In this step, you create an API key for Confluent Cloud. The Terraform scripts (that Bob generates) need this API key to provision the Confluent Cloud infrastructure.

1. Login to Confluent Cloud. From the Hamburger menu in the top right corner, click API Keys.

   

2. For this tutorial, choose My account. This API key will give access to your Confluent account with the same access as you.

   For production systems, you should choose Service account for more restricted access.

3. Cloud resource management for the scope of the API key, and then click Next. This option gives access to managed resources including Kafka and Apache Flink clusters within Confluent Cloud.

   

4. Give the API key a name and description, and then click Next.

5. Copy the key and save them somewhere safe so you can paste them in a future step and then click Complete.

Step 2. Create a Bob workspace and configure the Bob skill

In this step, you create a Bob workspace and configure the confluent-iac-terraform Bob skill. Bob skills are reusable instruction sets that teach Bob new workflows and specialized tasks. The confluent-iac-terraform skill defines the process for analyzing streaming use case requirements and generating complete, production-ready real-time streaming systems on Confluent Cloud using Infrastructure-as-Code (Terraform), Apache Flink SQL, and Python producers.

1. Open Bob IDE.

2. Click File > Open Folder.

3. Click New Folder, and name the folder fraud-confluent-terraform-bob. Then, click Open.

4. Create a new folder .bob to store all Bob related configurations.

5. Create a new subfolder .bob/skills to store the different skills.

6. Create a new subfolder .bob/skills/confluent-iac-terraform.

7. Download the SKILL.md from the Technology Building Blocks GitHub repo to this new folder.

   

Step 3. Create Terraform and Python scripts

In this step, you use the confluent-iac-terraform Bob skill to create the Terraform scripts to provision the Confluent infrastructure (Kafka Cluster, Apache Flink Cluster, topics, and Flink SQL statements) to implement the fraud detection system, and also create a Python script to send sample transactions to the topic to simulate a fraud use case and test the fraud detection system. You won’t actually ask Bob to use this specific skill, Bob will infer it based on the prompt.

1. In the Bob IDE, open the Bob chat interface.

2. Make sure that you are in Advanced mode to be able to access Bob skills.

3. In natural language, give Bob instructions to build the real-time banking fraud detection streaming system with Terraform and Confluent. Be very specific. For example, use this prompt in the Bob chat interface.

    Build a real-time banking fraud detection system using Confluent Cloud (Kafka + Flink) and Terraform.
   
    The system should stream banking transactions through Kafka topics, process them with Flink SQL to calculate 5-minute transaction counts per account and flag accounts with more than 10 transactions in a 5-minute window, and provision all infrastructure as code with Terraform.
   

   

4. Bob identified that this request is related to the confluent-iac-terraform skill, and asks you to approve activating the skill. Click Approve.

   

5. Bob creates the todo list which defines the main tasks that Bob will follow, mainly: analyzing requirements, generating Terraform infrastructure files, creating Flink SQL fraud detection logic, building a Python producer for sample transactions, writing comprehensive documentation, and validating all components.

   

6. Bob starts creating the Terraform infrastructure files and Python producer script. Review the generated files, and give Bob the approval until all components for the fraud detection system are created:

   • providers.tf: Configures the Confluent Cloud, Kafka, and Apache Flink providers with required versions and authentication.
   • variables.tf: Defines input variables for API credentials, cluster IDs, and environment configuration.
   • main.tf: Creates the core Confluent Cloud resources including Kafka topics, Flink compute pools, and fraud detection statements.
   • outputs.tf: Exports values like cluster endpoints, topic names, and resource IDs for reference.

   • produce_messages.py: Generates and publishes sample banking transaction messages to Kafka topics for testing the fraud detection system.

     Bob completes creating the different Terraform and Python scripts. In addition, Bob creates realistic sample banking transaction data in sample-transactions.json, and comprehensive documentation including:

   • README.md for project overview

   • SETUP.md for installation instructions
   • TESTING-APPROACH.md for validation procedures

   • A .gitignore file to exclude sensitive credentials and temporary files from version control.

     

7. Bob asks for approval to run validation commands: terraform init to download required provider plugins for Confluent, terraform validate to verify the configuration syntax and structure, and Python syntax validation commands to ensure the producer script is error-free. Click Run to execute these validation steps.

   

   Bob confirms that all creation and validation is completed successfully, with all 6 todo items marked as done: infrastructure files generated, Flink SQL logic created, Python producer built, documentation written, and all components validated through Terraform and Python syntax checks.

   

Step 4. Run the Terraform scripts to provision the Confluent infrastructure

In this step, you ask Bob to run the Terraform script to provision the Confluent Cloud infrastructure, which deploys the Kafka cluster, creates Flink compute pools, sets up SQL tables for fraud detection, and generates the necessary credentials for the Python producer.

1. In the Bob IDE chat interface, make sure that you are in Advanced mode to be able to access Bob skills.

2. Click Start New Task. It is a best practice to start a fresh task when moving to new stages as this clears Bob's context, reduces token usage, and ensures higher quality responses that are focused specifically on the infrastructure provisioning phase.

3. Bob created terraform.tfvars.example as a template file which shows the required configuration variables (Confluent Cloud API credentials, environment name, cluster name, region, cloud provider, and Flink settings). Copy it into a terraform.tfvars file, and update it to use your Confluent Cloud credentials (API key and API secret).

   

4. Create a .bobignore file on the workspace root folder to prevent Bob from reading or modifying sensitive files containing secrets and API keys. Include the following lines in this file.

    # Sensitive environment files
    python/.env
    .env
   
    # Terraform sensitive variables
    terraform/terraform.tfvars
   

   

5. In natural language, give Bob instructions to provision the Confluent infrastructure using the Terraform scripts that Bob created in the previous step.

    I have added my Confluent credentials, please run Terraform scripts.
   

6. Bob creates the todo list for deploying the fraud detection Confluent infrastructure, breaking down the Terraform deployment process into four steps: initialize Terraform (download providers), validate Terraform configuration (check syntax), apply Terraform to create infrastructure (provision resources), and verify deployment outputs (confirm successful creation). Click Approve to have Bob deploy the fraud detection Confluent infrastructure.

   

7. Bob has validated the Terraform configuration successfully, and now requests approval to provision the Confluent Cloud infrastructure. Click Run to execute the Terraform apply command. It will take a few minutes to create the different resources.

   If Bob tells you that “API credentials are invalid”, make sure that you updated the Confluent API credentials in the terraform.tfvars file.

   

8. Once Bob is done with applying Terraform to create the infrastructure, Bob requests approval to run terraform output to verify and display the deployment outputs, including cluster endpoints, topic names, and resource IDs that will be needed for the Python producer configuration. Click Run.

   

   Bob completes the verification activities.

   

Step 5. Verify the Confluent infrastructure in Confluent Cloud

Now it's time to check out the resources that you just provisioned. Confluent Cloud organizes resources in a hierarchical fashion: the top-level element is the organization, which contains 1 or more environments. Within a single environment, you have 1 Schema Registry instance, but you can host multiple Kafka clusters and Flink compute pools. While this hierarchy only scratches the surface of Confluent's capabilities, it provides the necessary starting point for navigating the console.

In your environment, there is a Kafka cluster and a Flink compute pool. The Terraform configuration also defined two Flink tables with a specific structure, which in turn created two corresponding Kafka topics. These topics are governed by a proper Data Contract schema linked via the Schema Registry, ensuring that your streaming pipeline is both structured and secure.

1. Open Confluent Cloud. The home page shows 1 environment, 1 Kafka cluster, 2 topics, and 1 Apache Flink compute pool.

   

2. Click Environments and then open the environment. It shows the Kafka cluster and Flink compute pool.

   

3. Click Flink on the left sidebar. You can see the Flink SQL statements in the Confluent Cloud console:

   • 2 CREATE TABLE statements that automatically create the underlying Kafka topics: 1 source table for incoming banking transactions and 1 sink table for detected fraudulent transactions

   • 1 INSERT INTO statement (running) that continuously analyzes the source stream and writes suspicious transactions to the sink running on 1 CFU (Confluent Flink Unit, the compute capacity unit for running Flink SQL jobs).

     

4. Click the INSERT INTO statement to understand more details. The Confluent SQL statement performs real-time fraud detection by creating 5-minute windows over the banking_transactions source table, grouping transactions by account_id and time window. It calculates the transaction count and total amount per window, then flags accounts as fraudulent (is_fraud = TRUE) when more than 10 transactions occur within a single 5-minute window. This is a common indicator of suspicious activity, doing a lot of transactions at small interval. The results are continuously written to the fraud_alerts sink table.

   

5. Click Clusters in the left menu to view the Kafka clusters.

6. Open the created Kafka cluster.

7. Click Topics in the left menu to view the created topics. These topics have been created by the Flink SQL Create Table statements. The banking_transactions topic will receive incoming transaction data from ATMs, internet banking, mobile apps, and point-of-sale systems, while the fraud_alerts topic stores the detected fraudulent transactions flagged by the real-time Flink SQL analysis.

   

8. Click one of the topics. The Overview tab provides immediate insights into the topic, including real-time traffic metrics and a narrowed data lineage view specific to that topic. You can also inspect the most recent messages and review the associated data contract.

   

Step 6. Simulate fraudulent banking transactions

In this step, you simulate fraudulent banking transactions by asking Bob to add sample transactions to the source Kafka topic.

1. In the Bob IDE chat interface, click Start New Task.

2. Instruct Bob in natural language to produce the message to the Kafka topic.

    Add sample banking transactions to the Kafka topic on Confluent.
   

   

3. Bob first checks that Terraform created the python .env file with Kafka connection credentials for the Python producer. Bob will validate that the .env file is already in-place. Click Run.

   

4. Bob installs the dependencies and runs the script to produce the sample transactions. Click Run.

   

   Bob confirms that all messages are delivered to the topic, and some of them are very frequent which indicates a fraud case.

   

5. Optionally, ask Bob to create a continuous Kafka producer that generates banking transactions every 0.5-1.5 seconds, with 85% going to fraud accounts and 15% to normal accounts. This allows you to observe real-time fraud detection as transactions flow continuously and triggers alert.

    Create a Python continuous Kafka producer for banking fraud detection. Generate transactions every 0.5-1.5 seconds with current timestamps. Use 5 fraud accounts (ACCOUNT-FRAUD-01 to 05) receiving 85% of transactions, and 5 normal accounts (ACCOUNT-10001 to 05) receiving 15%. Include varying transaction types and amounts. Use Confluent Kafka with Schema Registry. Display real-time stats. Run until Ctrl+C.
   

   

Step 7. Check for fraudulent cases in real time in Confluent Cloud

The Apache Flink integration in Confluent Cloud provides rich visual tools to introspect your data products from discovering data structures to building and running SQL statements in a powerful workspace.

1. Open Confluent Cloud, click Compute pools, and then open your Flink compute pool.

   

2. Click Open SQL workspace.

3. Write the following SQL statement, and then click Run. Make sure to update the table name based on what Terraform has created.

    SELECT * from `fraud_alerts`;
   

   

4. In real-time, Confluent detects the banking fraud. (You can connect to AI agents on watsonx Orchestrate to take actions in real-time, as described in this tutorial).

   

5. Directly query the table by clicking the Query with Flink button from the topic view. You can also see a histogram of the values for each column.

   

Step 8. Clean up

Once you are done, make sure that you clean up the Confluent resources. You can use Terraform destroy or just ask Bob to clean up the resources by this prompt “Hi Bob, please use Terraform destroy to clean up the Confluent resources”.

Summary and next steps

In this tutorial, you learned how to build a real-time banking fraud detection system using IBM Bob, Confluent Cloud (Apache Kafka and Apache Flink), and HashiCorp Terraform. You learned about infrastructure-as-code automation, real-time stream processing, and building an end-to-end streaming pipeline.

This is a summary of the tasks accomplished by Bob.

You can also explore how Bob helps creating MCP tools, and watsonx Orchestrate in this tutorial: Using IBM Bob to build watsonx Orchestrate agents and MCP tools.

Acknowledgments

This tutorial was produced as part of an EMEA initiative sponsored by Ralph Demuth, the Building Blocks initiative sponsored by Jason Gartner, and the IBM Open Innovation Community initiative to develop tutorials and learning content for integrating IBM Bob with other products.

The authors (Ahmed Azraq and Brice Leporini) appreciate the support of Frank Ketelaars, Carlo Appugliese, and Himangshu Mech for reviewing and contributing to the tutorial.