A series of video tutorials to build out, from a blank canvas, a properly-representative hybrid integration solution between on premises systems and applications on multiple clouds. This article and its hybrid integration solution goes beyond the basic demos often used by integration product vendors.
This article continues with a second article Hybrid Integration Part 2 â IBM Application Integration Suite in a representative hybrid cloud landscape, which explores how we can move the integration platform itself from IBM Integration Bus on premises to IBM Integration Bus on Cloud â a managed service for IIB available from the IBM Marketplace.
You don’t have to “google” too hard to find
hybrid integration demos showing various integration product vendors displaying their wares with whizz-bang 10-minute tool demos. These demos impress us with their speed and ease of use, but they ignore the reality of a hybrid, on premises to cloud (often multi-cloud) landscape.
Tool demos are just that… demos
Let me show you what I mean. These next two videos take you on a 10-minute journey of synchronizing on-premises data with a Software as a Service (SaaS) application; GMail for simplicity in this case:
In the first video, I use IBM Integration Bus for on-premises integration and event publication to the cloud.
In this second video showing the SaaS adaption one of my American colleagues demos use of IBM App Connect.
So what have we seen here?… We’ve seen that integration in the absence of a representative architectural landscape is as easy with IBM tools as it is with other vendors.
Representative hybrid integration Landscape
So let’s look at a more representative hybrid integration landscape.
In this typical landscape we have to consider integration in the context of multiple factors:
- Multiple SaaS providers in multiple clouds
- Cloud-based Platform as a Service (PaaS) environments for innovation and the rapid creation of new systems of engagement
- Socialization of published APIs to on-premises systems of record
- Proxying of those APIs for control/management/monetization
- Secure access between the clouds and on premises systems
The integration landscape above is a simplified view of the more complete Hybrid Integration Reference Architecture that is explored in detail by the following article by Kim Clark and Rob Nicholson:
However, for the purposes of the “hands on” hybrid integration journey that this article takes you on, the integration landscape above represents a solid starting point and allows me to explore and demonstrate (key press by key press) how we address the aspects of integration in the hybrid world with IBM products.
Building out a hybrid integration solution step by step
Ok, so go grab a nice cup of tea and get yourself comfortable because we are going to build out, from a blank canvas, the hybrid integration solution shown below. The scenario I have chosen is a simplified use case from the insurance industry (but it would be very easy to amend for customers in finance or or stock in retail for example). We have an integration requirement where a mainframe CICS system is creating a new member, that member needs name and address details logged to a distributed database. The insurance company in question is adopting a cloud strategy and has a number of SaaS endpoints that also need to be synchronized with the new member details.
- IBM Integration Bus v10 – IBM’s strategic integration technology and ESB
- IBM API Connect v5 – API Management, creation and socialization
- IBM App Connect – Lightweight SaaS Connectivity offering for adaption of SaaS to SaaS and SaaS to on-premise applications
- IBM Bluemix – IBM Platform as a Service offering
List of tutorials
The following set of instructional videos takes you through a total of about 2 hours of IBM Hybrid Integration tutorials to build the full end-to-end integration solution. When I’m not providing voice over and explaining the steps as I go it’s actually about a 1-hour build and test process. Depending on your level of familiarity with tools such as IBM Integration Bus (IIB) you may want to skip some of the early tutorials. For example, the first three videos talk to base on-premises integration between IBM MQ and DB2 using IIB v10. Some folks might want to start at the fourth video where I explore Restful API enablement of existing IIB integration services.
- Tutorial 1: Setting up the IIB tooling and runtime for integration with IBM MQ and Relational Database, DB2
- Tutorial 2: IIB – DFDL Tooling for modelling CICS fixed format data
- Tutorial 3: IIB – Creation of a CICS over MQ to RDBMS, DB2 database integration service
- Tutorial 4: IIB – RESTful enablement of the CICS to RDBMS integration service
- Tutorial 5: IIB – Testing local invocation of RESTful Interface to MQ
- Tutorial 6: IIB – Testing local invocation of the RESTful interface to RDBMS, DB2
- Tutorial 7: Secure Gateway – Configure and Test a secure gateway between on premises and PaaS environments, IBM Bluemix
- Tutorial 8: IBM Bluemix – Create a Node.js test application using Bluemix NodeRed
- Tutorial 9: IIB – Run the IIB IBM App Connect YAML and event enablement pattern
- Tutorial 10: IIB – Modify the generated IBM App Connect YAML file for IPaddress and port number
- Tutorial 11: IIB – Enable the IIB MQ to DB2 integration service for event publication to IBM App Connect
- Tutorial 12: IIB and App Connect- Test IIB event synchronization to Googlesheets via IBM App Connect
- Tutorial 13: IBM App Connect – Build an App Connect flow from scratch and retest event synchronization
- Tutorial 14: IBM App Connect – Create a new App Connect flow to subscribe to IIB and synchronize to SFDC
- Tutorial 15: IBM API Connect – Publish IIB APIs to API Connect and create an API proxy
- Tutorial 16: IIB – Exploring the IIB Debugger and Flow exerciser built in tools
- Want to build it out yourself?
- Want to run it yourself?
- Want to run it with out the MQ and DB2 requirement?
Tutorial 1: Setting up the IIB tooling and runtime for integration with IBM MQ and Relational Database, DB2
In this tutorial we complete some simple steps to set up the IIB toolkit and run time for integration with IBM MQ and Relational Databases. It demonstrates how IIB can be used directly against relational databases such as DB2, for testing and database manipulation instead of using tools supplied by that RDBMS (Relational Database Management System) vendor. It also shows how you can use configurable services to abstract the specifics of a particular database instance from the artifacts that we are developing IIB. This helps in support of a smooth and agile integration delivery lifecycle.
Next we look at data modelling in IIB…
Tutorial 2: IIB – DFDL tools for modelling CICS fixed format data
In our scenario at its base level of on-premise systems integration we are integrating a CICS system with an RDBMS. Therefore, we need to deal with the specifics of legacy (non XML, non JSON) data. In this tutorial we explore the IIB Data Format Definition Language (DFDL) tools that allows us to model, parse, and serialize any kind of data without code.
Having set up our environment and modeled our CICS fixed-format data we next create the IIB CICS to RDBMS integration service…
Tutorial 3: IIB – Creation of a CICS over MQ to RDBMS, DB2 database integration service
This tutorial takes us through the step-by-step creation of a simple integration service in IIB, to take a new member based on a CICS fixed format message from IBM MQ and insert it into an RDBMS, in this case DB2. We keep it simple and build an integration that is two simple steps (get from MQ and insert into Database) and use a graphical mapping approach to avoid the need to drop into code.
This base integration service provides the starting point for our hybrid integration journey as we move onto the next tutorials.
Next up is RESTful service enablement…
Tutorial 4: IIB – RESTful enablement of the CICS to RDBMS integration service
In this tutorial we add a RESTful API interface to our CICS to RDBMS service.
So far our integration service has enabled a CICS system that is creating new members to ensure that each new member is logged into our master member database.
What if our organization wanted to enable other, new applications to create new members? Adding a RESTful memberDBStore interface with an addMember operation will allow such new applications and systems to rapid integrate with what was traditionally a legacy system.
Let’s create an interface on IIB that offers Restful access to our IBM MQ input queue.
Next we’ll unit test our new RESTful memberDBStore (add member) interface as far as our MQ queue…
Tutorial 5: IIB – Testing local invocation of RESTful Interface to MQ
In this tutorial, we turn off the base CICS MQ to RDBMS integration service and test that driving our RESTful interface locally does correctly result in a CICS fixed-format message landing on our MQ queue.
Next, we’ll test that driving our RESTful interface correctly results in a new member being logged in the database.
Tutorial 6: IIB – Testing local invocation of the RESTful interface to RDBMS, DB2
Let’s turn on the CICS MQ to RDBMS integration service, then retest the local invocation of our RESTful interface, and then check the results in the database.
So, we have enabled our CICS MQ to RDBMS service with a RESTful API and it can be accessed locally and directly on premises. But, we live in a hybrid world so let’s look at how we can quickly and securely give our RESTful service a public IP address that is consumable in the cloud…
Tutorial 7: Secure Gateway – Configure and Test a secure gateway between on premises and PaaS environments, IBM Bluemix
In this tutorial we exploit IBM’s Platform as a Service (PaaS) offering, Bluemix, and in particular it’s Secure Gateway service to set up a secure web socket tunnel between the cloud and our on-premises services.
We use the Secure Gateway Client in a Docker container on premises, but there is an installable client or the option to leverage Virtual or physical Datapower appliances. After configuring, we retest our RESTful memberDBStore interface to add new members to the on-premises database through a secured, public IP address in the cloud.
Good. So now we look at developing new Systems of Engagement (SoE) that can call upon our add member RESTful service. By using a PaaS environment such as Bluemix, we can quickly build and test with Ruby in Rails, PHP, or Node.js for example.
Let’s do something using NodeRed in Bluemix…
Tutorial 8: IBM Bluemix – Create a Node.js test application using Bluemix NodeRed
The IBM Bluemix Platform as a Service is enabled for instant run times such as Java and Node.js and allows us to rapidly provision applications for those run times. The NodeRed service in Bluemix enabled me to create and test a Node.js application to drive the addMember Restful interface to our on-premises database through the Secure Gateway. NodeRed lets me construct graphically the application (which in our simple example takes the form of a test harness), avoiding the need for a Node.js novice like myself to work with code.
Ok, so now we have proven that the real developers out there in our organization can start building applications, microservices, and SoEs that use our addMember Restful interface. However, we don’t want to go public yet just yet. (We’ll do that later using IBM API Connect to socialize the availability of a Restful memberDBStore API with addMember operation and create a management/monitoring API proxy).
First, I have another integration task I’d like to perform. I want to enable my base CICS MQ to Database integration service for events, such that IIB will publish ‘add member’ events each time a new member is logged in the database. (Regardless of whether this is through direct entry from CICS or through new applications using the Restful interface from the Cloud.)
Tutorial 9: IIB – Run the IIB IBM App Connect YAML and event enablement pattern
IIB features a capability called patterns-based development. This feature allows typical integration tasks to be performed automatically by tools instead of relying on the integration developer to create or wire together integration artifacts by hand.
Patterns-based development takes pre-defined, repeatable integration services, flows, and logic, and exposes only the points of variation to the integration developer. As a result you get best-practice creation of integration artifacts every time, eliminating finger press errors, and so on.
Let’s go ahead and run the IBM App Connect Custom App pattern:
The IBM App Connect Custom App pattern has created a YAML file to describe our add Member to database service as an event. It has also generated the subflows for us to wire into our CICS MQ to Database integration service in order to event enable it for IBM App Connect.
Tutorial 10: IIB – Modify the generated IBM App Connect YAML file for IPaddress and port number
The version of the IBM App Connect Custom App pattern for IIB that I used in the previous tutorial is an early version that does not complete all the customization of the YAML file. Future versions will be more comprehensive. Therefore, at this point in time there are a few edits to the YAML file we must complete by hand. This tutorial works through the manual steps to ready the YAML file for consumption by IBM App Connect.
A completed YAML file is representative of a “product” in IBM App Connect. Therefore, this IIB generation process allows us to describe any system or application sitting behind IIB as a product that can be consumed by IBM App Connect.
Next lets complete the event enablement of the CICS MQ to Database service…
Tutorial 11: IIB – Enable the IIB MQ to DB2 integration service for event publication to IBM App Connect
In this tutorial we wire the generated, event publication subflow into the CICS MQ to Database integration service, and create a graphical map to populate the contents of the event to be published when a new member is added to the database.
Good. So that’s our CICS MQ to Database integration service event enabled for IBM App Connect, and ready to be driven as a RESTful interface from the cloud.
Tutorial 12: IIB and App Connect- Test IIB event synchronization to Googlesheets via IBM App Connect
In this tutorial we perform our first end-to-end test. We drive the CICS MQ to Database integration service from NodeRed in Bluemix, check that the member is correctly added to the database, and check that our event is published by IIB and consumed by IBM App Connect for synchronization with Google Sheets. For this test I have already created an IBM App Connect flow for Member Database on premises to Google Sheets, and it is up and running in the IBM Cloud.
That worked a treat. So let’s see how we create the custom application in App Connect based on the YAML file and build a flow to integrate the event from on premises with a SaaS application.
Tutorial 13: IBM App Connect – Build an App Connect flow from scratch and retest event synchronization
IBM App Connect is a very simple integration tool to enable integration with typical SaaS applications. It can be extended by importing a YAML file to describe custom applications (in our case, our on premises member database). IBM App Connect can be used by Central IT or Line of Business professionals with a lower level of technical background to achieve such integration tasks.
This tutorial shows you just how easy IBM App Connect is to use.
Imagine Central IT had used IBM App Connect to enable many of the on-premise systems interfaces on IIB, and produced YAML files that describe each of those on-premises systems as applications in IBM App Connect. This would provide easy access for Line of Business professionals to a wide range of on-premises data, and be able to make that data available on-demand, without further work by Central IT to the SaaS applications they use.
Tutorial 14: IBM App Connect – Create a new App Connect flow to subscribe to IIB and synchronize to SFDC
In this tutorial, I’ll introduce my good friend and colleague, Peter Jessup, to build a second IBM App Connect flow and have it subscribe to new member database events, but this time we’ll have it add the event data to Salesforce.com:
Notice, because of the pub/sub event style that was implemented, there was no further integration work to be done outside of App Connect. New users with new SaaS apps can simply subscribe.
Now let’s take a moment to recap what we have achieved so far. We have completed the following:
- Integrated a CICS application with a Relational Database on premises
- Created a Restful interface that we made available securely in the cloud and invoked from a Node.js test application.
- The event-enabled integration service on premises is publishing events that capture the creation of a new member in the database
- IBM App Connect has two subscribing flows that synchronize the on premises database content with SaaS applications Google Sheets and Salesforce.com
And all that through configuration only, not one line of code written.
Tutorial 15: IBM API Connect – Publish IIB APIs to API Connect and create an API proxy
Earlier in the article I mentioned that we use IBM API connect to socialize the RESTful interface (that we have created for our memberDBStore on premises) with our developer community and to create a API management/control proxy. Let’s look at that next.
In this tutorial, Peter Jessup takes you through the process of pushing our API into IBM API Connect and testing it from there.
So, IBM API Connect covers the full gambit of API management, socialize, manage and create through its graphical Node.js framework for orchestration of RESTful interfaces into APIs.
Peter is on a roll now, so lets have him wrap up for us with a quick look at some of the IBM Integration Bus test tools, as we haven’t seen them in action whilst building out this hybrid integration solution…
Tutorial 16: IIB – Exploring the IIB Debugger and Flow exerciser built in tools
In this tutorial, we take a quick look at the IIB Flow Exerciser and IIB Debugging tools:
So there you have it.
Two 3rd-party public clouds (with SFDC and Google Sheets), one IBM public cloud (running IBM App Connect), one public Platform as a Service (Bluemix with Secure Gateway, NodeRed and API Connect services), an on-premises VM Image, only an hour of configuration, and zero lines of code written.
Want to build it out yourself?
All the collateral (end state) and documentation is available from GitHub https://github.com/DAVEXACOM/IBMHybridIntegrationDemo
Want to run it yourself?
To run this set of tutorials yourself, you need the following resources:
- An IBM App Connect license available through the IBM Marketplace (Free trial available)
- Google Sheets account
- Salesforce.com developer account
- IBM Bluemix account (Free trial available)
Want to run it with out the MQ and DB2 requirement?
To test synchronization with SaaS apps through IBM App Connect using the memberDBStore Restful API without the DB2 requirement and only placing messages on MQ as an option, make the following modifications to the memberDBStore, postMember subflow:
- Delete the Pass Through node
- Replace the Pass Through node with a Mapping Node and WebHookOutput subflow (you need to add a project reference)
- Configure the WebHookOutput and it’s subflow identically to how it is configured in CICSMemberToDBStore Application
- Wire the Mapping node and WebHookOut subflow between the Input and Output terminals
- Optionally remove the CICS.MEMBER.OUT node and save your work
- Stop/undeploy CICSMembertoDBStore application and original memberDBStore from the runtime
- Redeploy the new version of the memberDBStore RestAPI
The updated Rest API and Mapping are shown below.
All the collateral (end state) and documentation for the tutorials in this article is available from GitHub https://github.com/DAVEXACOM/IBMHybridIntegrationDemo
Explore other tutorials in Hybrid Integration Part 2 â IBM Application Integration Suite in a representative hybrid cloud landscape, which explores how we can move the integration platform itself from IBM Integration Bus on premises to IBM Integration Bus on Cloud â a managed service for IIB available from the IBM Marketplace.