Create a data lakehouse and integrate with Amazon Web Services using IBM watsonx.data on AWS

By connecting and associating data from existing AWS and IBM data sources with the query engine in IBM watsonx.data, you can gain a powerful and unified view of your data. That view will provide faster and more comprehensive insights. This integration can be achieved without the cost and complexity of duplicating and moving data across different environments.

This tutorial gets you started on IBM watsonx.data on Amazon Web Services (AWS). It will guide you through the steps to connect, explore, and ingest AWS data sources, including Amazon S3, Amazon RDS, Amazon Aurora PostgreSQL databases, and tables managed by AWS Glue.

After completing the tutorial, you will be able to create a data lakehouse on IBM watsonx.data on AWS and connect to various AWS services. We've also included included links and demos for you to dive deeper into IBM watsonx.data on AWS.

Prerequisites

• An AWS account to access AWS services
• IBM watsonx.data SaaS subscription (URL and credentials)
• Existing data sources on AWS (for example, Amazon S3, Amazon RDS, Amazon Aurora PostgreSQL) or other cloud services that you want to integrate with IBM watsonx.data

Steps

To connect IBM watsonx.data to AWS and explore and ingest AWS data sources, complete the following steps:

Connect AWS data sources in IBM watsonx.data

1. Go to IBM watsonx.data and log in using your credentials .
2. To define and associate your infrastructure components, such as engines, catalogs, and buckets, click the hamburger menu and select Infrastructure Manager.
3. In the Infrastructure Manager window, click Add Component and select Create Engine.
4. In the Create Engine window, provide the required details, including engine type, display name, size, and (optionally) linking a catalog. After the engine is provisioned, it will be visible in the Infrastructure manager window, along with any associated catalogs.
5. Configure connections to your AWS data sources, such as Amazon S3 buckets and databases, using the appropriate watsonx.data connectors, and link them to the previously provisioned engine.
   1. To add an Amazon S3 bucket to watsonx.data, click the Add component list and select Add bucket. In the Add bucket window, enter the bucket details, including type, name, endpoint, access key, secret key, and catalog name. When you are finished, click Add to include the bucket in a catalog.
   2. To integrate AWS database services such as Amazon RDS or Amazon Aurora PostgreSQL into watsonx.data, click the Add Component list and select Add Database.
   3. Select the relevant database type and provide the necessary details. Note: In watsonx.data v1.0.3, there are no AWS-specific database connectors. However, you can select the appropriate generic connector. For example, you can select PostgreSQL for Amazon Aurora PostgreSQL and provide the necessary details.

Explore, curate, and ingest data from an AWS data source

1. On the IBM watsonx.data home page, click the hamburger menu and select Data manager. The Data manager window opens.
2. In the Data manager window, click the Data objects tab and then select an engine to view the catalogs associated with it.
3. In the Catalogs associated list, click to expand a catalog and then select an associated table. In the table pane, four tabs are displayed (see the image that follows):
   • Table schema: Shows data types and additional metadata information
   • Time travel: Enables rollback to a specific snapshot for historical reference
   • Data sample: Provides a view of sample rows within the data set
   • DDL: Presents the data definition language for insight into the structure of the table
4. In the table pane, click Table schema. Select a table to view the schema and sample data.
5. Optional: To roll back data to a particular snapshot, click Time travel and select the snapshot you want to roll back to.

Run a query against AWS data

1. On the watsonx.data homepage, click the hamburger menu and select SQL. The Query workspace window opens. It includes the following components:
   • Data objects: View the engines, catalogs, schemas, tables, and columns
   • Engine: Select an engine and view the associated catalogs
   • Saved queries: View any saved queries
   • Worksheet Write SQL queries
2. From the Engine list, select an engine and then select the specific catalog, schema, table, or column where you want to run the query.
3. In the Worksheet, enter the SQL query against the tables. When you are finished, click Run to run the query.
4. Optional: To save your query, click the Save icon to in the menu bar.

Connect to AWS Glue database and tables

The AWS Glue Data Catalog is a centralized metadata repository for data assets across various data sources. Note: The steps that follow are a general guide only. For specific instructions, see the current AWS documentation.

1. Log in to your AWS account or create a free account.
2. On the AWS Console page, search for AWS Glue .
3. In the AWS Glue console, click Data catalog > Databases. In the Databases window, click Add database.
4. In the Create a database window, enter a name for the database and click Create database.
5. To create a table using the AWS Glue console, in the left menu, click Tables > Create table.
6. Enter the table properties, such as name and database, select S3 as the data store, and add the data store path.
7. Click Next > Next > Create to create the table.
8. To connect to the AWS Glue Data Catalog in watsonx.data, on the watsonx.data home page, click the hamburger menu and click Add component > Add database
9. From the Database definition list, select Custom.
10. In the Add database window, add the custom database connection properties from AWS Glue. When you have added all required properties, click Add.

    shell
    hive.metastore=glue
    hive.metastore.glue.region=<aws glue region>
    hive.metastore.glue.aws-access-key=<aws access key>
    hive.metastore.glue.aws-secret-key=<aws secret key>
    hive.s3.aws-access-key=<aws access key>
    hive.s3.aws-secret-key=<aws secret key>
    

    
11. On the watsonx.data home page, click the hamburger menu and then click Infrastructure manager to verify that AWS Glue has been added under Databases.
12. Click the hamburger menu and then click Data manager. If you have successfully established a connection with AWS Glue, you will see the databases managed by the AWS Glue Data Catalog in the Data objects list. On the Data objects tab, select the AWS Glue catalog and confirm that the databases and tables from AWS Glue are listed.

Summary and next steps

IBM watsonx.data is a data management solution for collecting, storing, querying, and analyzing all your enterprise data with a single unified data platform. In this tutorial, you have learned how to create a data lakehouse and connect, store, query, and analyze data from different AWS data sources. With these integrations, AWS and IBM are working together to deliver highly available and secure analytics and AI workloads within a unified and governed ecosystem.

To learn more about watsonx.data and the IBM / AWS partnership, see the following resources:

• Read the IBM watsonx.data on AWS overview.
• View the watsonx.data AWS Marketplace listing and deployment template.
• Begin your watsonx.data on AWS trial experience.
• Learn more about watsonx on AWS.
• Discover IBM watsonx, the portfolio of AI products for the enterprise.
• Explore the IBM Developer Amazon Web Services (AWS) hub to learn more about IBM and AWS.
• Contact your IBM representative to get a product overview, trial experience, demo, or a deep dive.

IBM | AWS partnership

With Amazon Web Services (AWS) and IBM, unleash the transformative value of generative AI in your business with greater speed, scale and trust. The IBM, AWS, and Red Hat partnership brings a unique combination of leading enterprise AI, cloud, infrastructure, and open source technologies delivered with deep IBM consulting expertise. This enables companies to quickly and responsibly scale AI workloads using a comprehensive stack of generative AI, composed of Amazon Bedrock and IBM watsonx running on AWS Cloud and across hybrid cloud environments.

Learn more about the IBM / AWS partnership.