Skip the Introduction? Go straight to Tutorial 1

A series of video tutorials to build out, from a blank canvas, a properly-representative hybrid integration solution between on premises systems and applications on multiple clouds. This article and its hybrid integration solution goes beyond the basic demos often used by integration product vendors.

This article continues with a second article Hybrid Integration Part 2 – IBM Application Integration Suite in a representative hybrid cloud landscape, which explores how we can move the integration platform itself from IBM Integration Bus on premises to IBM Integration Bus on Cloud – a managed service for IIB available from the IBM Marketplace.

Introduction

You don’t have to “google” too hard to find hybrid integration demos showing various integration product vendors displaying their wares with whizz-bang 10-minute tool demos. These demos impress us with their speed and ease of use, but they ignore the reality of a hybrid, on premises to cloud (often multi-cloud) landscape.

Tool demos are just that… demos

Let me show you what I mean. These next two videos take you on a 10-minute journey of synchronizing on-premises data with a Software as a Service (SaaS) application; GMail for simplicity in this case:

ToolingDemo Hybrid Integration

In the first video, I use IBM Integration Bus for on-premises integration and event publication to the cloud.

In this second video showing the SaaS adaption one of my American colleagues demos use of IBM App Connect.

So what have we seen here?… We’ve seen that integration in the absence of a representative architectural landscape is as easy with IBM tools as it is with other vendors.

Representative hybrid integration Landscape

So let’s look at a more representative hybrid integration landscape.

Hybrid integration representative landscape

In this typical landscape we have to consider integration in the context of multiple factors:

  • Multiple SaaS providers in multiple clouds
  • Cloud-based Platform as a Service (PaaS) environments for innovation and the rapid creation of new systems of engagement
  • Socialization of published APIs to on-premises systems of record
  • Proxying of those APIs for control/management/monetization
  • Secure access between the clouds and on premises systems

The integration landscape above is a simplified view of the more complete Hybrid Integration Reference Architecture that is explored in detail by the following article by Kim Clark and Rob Nicholson:

http://www.ibm.com/developerworks/library/mw-1606-clark-trs/index.html

However, for the purposes of the “hands on” hybrid integration journey that this article takes you on, the integration landscape above represents a solid starting point and allows me to explore and demonstrate (key press by key press) how we address the aspects of integration in the hybrid world with IBM products.

Back to top

 

Building out a hybrid integration solution step by step

Ok, so go grab a nice cup of tea and get yourself comfortable because we are going to build out, from a blank canvas, the hybrid integration solution shown below. The scenario I have chosen is a simplified use case from the insurance industry (but it would be very easy to amend for customers in finance or or stock in retail for example). We have an integration requirement where a mainframe CICS system is creating a new member, that member needs name and address details logged to a distributed database. The insurance company in question is adopting a cloud strategy and has a number of SaaS endpoints that also need to be synchronized with the new member details.

Hybrid Integration End to End demo

 

Products Used

  • IBM Integration Bus v10 – IBM’s strategic integration technology and ESB
  • IBM API Connect v5 – API Management, creation and socialization
  • IBM App Connect – Lightweight SaaS Connectivity offering for adaption of SaaS to SaaS and SaaS to on-premise applications
  • IBM Bluemix – IBM Platform as a Service offering


List of tutorials

The following set of instructional videos takes you through a total of about 2 hours of IBM Hybrid Integration tutorials to build the full end-to-end integration solution. When I’m not providing voice over and explaining the steps as I go it’s actually about a 1-hour build and test process. Depending on your level of familiarity with tools such as IBM Integration Bus (IIB) you may want to skip some of the early tutorials. For example, the first three videos talk to base on-premises integration between IBM MQ and DB2 using IIB v10. Some folks might want to start at the fourth video where I explore Restful API enablement of existing IIB integration services.

  1. Tutorial 1: Setting up the IIB tooling and runtime for integration with IBM MQ and Relational Database, DB2
  2. Tutorial 2: IIB – DFDL Tooling for modelling CICS fixed format data
  3. Tutorial 3: IIB – Creation of a CICS over MQ to RDBMS, DB2 database integration service
  4. Tutorial 4: IIB – RESTful enablement of the CICS to RDBMS integration service
  5. Tutorial 5: IIB – Testing local invocation of RESTful Interface to MQ
  6. Tutorial 6: IIB – Testing local invocation of the RESTful interface to RDBMS, DB2
  7. Tutorial 7: Secure Gateway – Configure and Test a secure gateway between on premises and PaaS environments, IBM Bluemix
  8. Tutorial 8: IBM Bluemix – Create a Node.js test application using Bluemix NodeRed
  9. Tutorial 9: IIB – Run the IIB IBM App Connect YAML and event enablement pattern
  10. Tutorial 10: IIB – Modify the generated IBM App Connect YAML file for IPaddress and port number
  11. Tutorial 11: IIB – Enable the IIB MQ to DB2 integration service for event publication to IBM App Connect
  12. Tutorial 12: IIB and App Connect- Test IIB event synchronization to Googlesheets via IBM App Connect
  13. Tutorial 13: IBM App Connect – Build an App Connect flow from scratch and retest event synchronization
  14. Tutorial 14: IBM App Connect – Create a new App Connect flow to subscribe to IIB and synchronize to SFDC
  15. Tutorial 15: IBM API Connect – Publish IIB APIs to API Connect and create an API proxy
  16. Tutorial 16: IIB – Exploring the IIB Debugger and Flow exerciser built in tools
  17. Conclusion
  18. Want to build it out yourself?
  19. Want to run it yourself?
  20. Want to run it with out the MQ and DB2 requirement?

Back to tutorial list | Back to top

 

Tutorial 1: Setting up the IIB tooling and runtime for integration with IBM MQ and Relational Database, DB2

In this tutorial we complete some simple steps to set up the IIB toolkit and run time for integration with IBM MQ and Relational Databases. It demonstrates how IIB can be used directly against relational databases such as DB2, for testing and database manipulation instead of using tools supplied by that RDBMS (Relational Database Management System) vendor. It also shows how you can use configurable services to abstract the specifics of a particular database instance from the artifacts that we are developing IIB. This helps in support of a smooth and agile integration delivery lifecycle.

Next we look at data modelling in IIB…

Back to tutorial list | Back to top

 

Tutorial 2: IIB – DFDL tools for modelling CICS fixed format data

In our scenario at its base level of on-premise systems integration we are integrating a CICS system with an RDBMS. Therefore, we need to deal with the specifics of legacy (non XML, non JSON) data. In this tutorial we explore the IIB Data Format Definition Language (DFDL) tools that allows us to model, parse, and serialize any kind of data without code.

Having set up our environment and modeled our CICS fixed-format data we next create the IIB CICS to RDBMS integration service…

Back to tutorial list | Back to top

 

Tutorial 3: IIB – Creation of a CICS over MQ to RDBMS, DB2 database integration service

This tutorial takes us through the step-by-step creation of a simple integration service in IIB, to take a new member based on a CICS fixed format message from IBM MQ and insert it into an RDBMS, in this case DB2. We keep it simple and build an integration that is two simple steps (get from MQ and insert into Database) and use a graphical mapping approach to avoid the need to drop into code.

This base integration service provides the starting point for our hybrid integration journey as we move onto the next tutorials.

Next up is RESTful service enablement…

Back to tutorial list | Back to top

 

Tutorial 4: IIB – RESTful enablement of the CICS to RDBMS integration service

In this tutorial we add a RESTful API interface to our CICS to RDBMS service.

So far our integration service has enabled a CICS system that is creating new members to ensure that each new member is logged into our master member database.

What if our organization wanted to enable other, new applications to create new members? Adding a RESTful memberDBStore interface with an addMember operation will allow such new applications and systems to rapid integrate with what was traditionally a legacy system.

Let’s create an interface on IIB that offers Restful access to our IBM MQ input queue.

Next we’ll unit test our new RESTful memberDBStore (add member) interface as far as our MQ queue…

Back to tutorial list | Back to top

 

Tutorial 5: IIB – Testing local invocation of RESTful Interface to MQ

In this tutorial, we turn off the base CICS MQ to RDBMS integration service and test that driving our RESTful interface locally does correctly result in a CICS fixed-format message landing on our MQ queue.

Next, we’ll test that driving our RESTful interface correctly results in a new member being logged in the database.

Back to tutorial list | Back to top

 

Tutorial 6: IIB – Testing local invocation of the RESTful interface to RDBMS, DB2

Let’s turn on the CICS MQ to RDBMS integration service, then retest the local invocation of our RESTful interface, and then check the results in the database.

So, we have enabled our CICS MQ to RDBMS service with a RESTful API and it can be accessed locally and directly on premises. But, we live in a hybrid world so let’s look at how we can quickly and securely give our RESTful service a public IP address that is consumable in the cloud…

Back to tutorial list | Back to top

 

Tutorial 7: Secure Gateway – Configure and Test a secure gateway between on premises and PaaS environments, IBM Bluemix

In this tutorial we exploit IBM’s Platform as a Service (PaaS) offering, Bluemix, and in particular it’s Secure Gateway service to set up a secure web socket tunnel between the cloud and our on-premises services.

We use the Secure Gateway Client in a Docker container on premises, but there is an installable client or the option to leverage Virtual or physical Datapower appliances. After configuring, we retest our RESTful memberDBStore interface to add new members to the on-premises database through a secured, public IP address in the cloud.

Good. So now we look at developing new Systems of Engagement (SoE) that can call upon our add member RESTful service. By using a PaaS environment such as Bluemix, we can quickly build and test with Ruby in Rails, PHP, or Node.js for example.

Let’s do something using NodeRed in Bluemix…

Back to tutorial list | Back to top

 

Tutorial 8: IBM Bluemix – Create a Node.js test application using Bluemix NodeRed

The IBM Bluemix Platform as a Service is enabled for instant run times such as Java and Node.js and allows us to rapidly provision applications for those run times. The NodeRed service in Bluemix enabled me to create and test a Node.js application to drive the addMember Restful interface to our on-premises database through the Secure Gateway. NodeRed lets me construct graphically the application (which in our simple example takes the form of a test harness), avoiding the need for a Node.js novice like myself to work with code.

Ok, so now we have proven that the real developers out there in our organization can start building applications, microservices, and SoEs that use our addMember Restful interface. However, we don’t want to go public yet just yet. (We’ll do that later using IBM API Connect to socialize the availability of a Restful memberDBStore API with addMember operation and create a management/monitoring API proxy).

First, I have another integration task I’d like to perform. I want to enable my base CICS MQ to Database integration service for events, such that IIB will publish ‘add member’ events each time a new member is logged in the database. (Regardless of whether this is through direct entry from CICS or through new applications using the Restful interface from the Cloud.)

Back to tutorial list | Back to top

 

Tutorial 9: IIB – Run the IIB IBM App Connect YAML and event enablement pattern

IIB features a capability called patterns-based development. This feature allows typical integration tasks to be performed automatically by tools instead of relying on the integration developer to create or wire together integration artifacts by hand.

Patterns-based development takes pre-defined, repeatable integration services, flows, and logic, and exposes only the points of variation to the integration developer. As a result you get best-practice creation of integration artifacts every time, eliminating finger press errors, and so on.

Let’s go ahead and run the IBM App Connect Custom App pattern:

The IBM App Connect Custom App pattern has created a YAML file to describe our add Member to database service as an event. It has also generated the subflows for us to wire into our CICS MQ to Database integration service in order to event enable it for IBM App Connect.

Back to tutorial list | Back to top

 

Tutorial 10: IIB – Modify the generated IBM App Connect YAML file for IPaddress and port number

The version of the IBM App Connect Custom App pattern for IIB that I used in the previous tutorial is an early version that does not complete all the customization of the YAML file. Future versions will be more comprehensive. Therefore, at this point in time there are a few edits to the YAML file we must complete by hand. This tutorial works through the manual steps to ready the YAML file for consumption by IBM App Connect.

A completed YAML file is representative of a “product” in IBM App Connect. Therefore, this IIB generation process allows us to describe any system or application sitting behind IIB as a product that can be consumed by IBM App Connect.

Next lets complete the event enablement of the CICS MQ to Database service…

Back to tutorial list | Back to top

 

Tutorial 11: IIB – Enable the IIB MQ to DB2 integration service for event publication to IBM App Connect

In this tutorial we wire the generated, event publication subflow into the CICS MQ to Database integration service, and create a graphical map to populate the contents of the event to be published when a new member is added to the database.

Good. So that’s our CICS MQ to Database integration service event enabled for IBM App Connect, and ready to be driven as a RESTful interface from the cloud.

Back to tutorial list | Back to top

 

Tutorial 12: IIB and App Connect- Test IIB event synchronization to Googlesheets via IBM App Connect

In this tutorial we perform our first end-to-end test. We drive the CICS MQ to Database integration service from NodeRed in Bluemix, check that the member is correctly added to the database, and check that our event is published by IIB and consumed by IBM App Connect for synchronization with Google Sheets. For this test I have already created an IBM App Connect flow for Member Database on premises to Google Sheets, and it is up and running in the IBM Cloud.

That worked a treat. So let’s see how we create the custom application in App Connect based on the YAML file and build a flow to integrate the event from on premises with a SaaS application.

Back to tutorial list | Back to top

 

Tutorial 13: IBM App Connect – Build an App Connect flow from scratch and retest event synchronization

IBM App Connect is a very simple integration tool to enable integration with typical SaaS applications. It can be extended by importing a YAML file to describe custom applications (in our case, our on premises member database). IBM App Connect can be used by Central IT or Line of Business professionals with a lower level of technical background to achieve such integration tasks.

This tutorial shows you just how easy IBM App Connect is to use.

Imagine Central IT had used IBM App Connect to enable many of the on-premise systems interfaces on IIB, and produced YAML files that describe each of those on-premises systems as applications in IBM App Connect. This would provide easy access for Line of Business professionals to a wide range of on-premises data, and be able to make that data available on-demand, without further work by Central IT to the SaaS applications they use.

Back to tutorial list | Back to top

 

Tutorial 14: IBM App Connect – Create a new App Connect flow to subscribe to IIB and synchronize to SFDC

In this tutorial, I’ll introduce my good friend and colleague, Peter Jessup, to build a second IBM App Connect flow and have it subscribe to new member database events, but this time we’ll have it add the event data to Salesforce.com:

Notice, because of the pub/sub event style that was implemented, there was no further integration work to be done outside of App Connect. New users with new SaaS apps can simply subscribe.

Recap

Now let’s take a moment to recap what we have achieved so far. We have completed the following:

  1. Integrated a CICS application with a Relational Database on premises
  2. Created a Restful interface that we made available securely in the cloud and invoked from a Node.js test application.
  3. The event-enabled integration service on premises is publishing events that capture the creation of a new member in the database
  4. IBM App Connect has two subscribing flows that synchronize the on premises database content with SaaS applications Google Sheets and Salesforce.com

And all that through configuration only, not one line of code written.

Back to tutorial list | Back to top

 

Tutorial 15: IBM API Connect – Publish IIB APIs to API Connect and create an API proxy

Earlier in the article I mentioned that we use IBM API connect to socialize the RESTful interface (that we have created for our memberDBStore on premises) with our developer community and to create a API management/control proxy. Let’s look at that next.

In this tutorial, Peter Jessup takes you through the process of pushing our API into IBM API Connect and testing it from there.

So, IBM API Connect covers the full gambit of API management, socialize, manage and create through its graphical Node.js framework for orchestration of RESTful interfaces into APIs.

Peter is on a roll now, so lets have him wrap up for us with a quick look at some of the IBM Integration Bus test tools, as we haven’t seen them in action whilst building out this hybrid integration solution…

Back to tutorial list | Back to top

 

Tutorial 16: IIB – Exploring the IIB Debugger and Flow exerciser built in tools

In this tutorial, we take a quick look at the IIB Flow Exerciser and IIB Debugging tools:

Back to tutorial list | Back to top

 

Conclusion

So there you have it.

Two 3rd-party public clouds (with SFDC and Google Sheets), one IBM public cloud (running IBM App Connect), one public Platform as a Service (Bluemix with Secure Gateway, NodeRed and API Connect services), an on-premises VM Image, only an hour of configuration, and zero lines of code written.

Back to tutorial list | Back to top

 

Want to build it out yourself?

All the collateral (end state) and documentation is available from GitHub https://github.com/DAVEXACOM/IBMHybridIntegrationDemo

Back to tutorial list | Back to top

 

Want to run it yourself?

To run this set of tutorials yourself, you need the following resources:

  1. An IBM App Connect license available through the IBM Marketplace (Free trial available)
  2. Google Sheets account
  3. Salesforce.com developer account
  4. IBM Bluemix account (Free trial available)

Back to tutorial list | Back to top

 

Want to run it with out the MQ and DB2 requirement?

To test synchronization with SaaS apps through IBM App Connect using the memberDBStore Restful API without the DB2 requirement and only placing messages on MQ as an option, make the following modifications to the memberDBStore, postMember subflow:

  1. Delete the Pass Through node
  2. Replace the Pass Through node with a Mapping Node and WebHookOutput subflow (you need to add a project reference)
  3. Configure the WebHookOutput and it’s subflow identically to how it is configured in CICSMemberToDBStore Application
  4. Wire the Mapping node and WebHookOut subflow between the Input and Output terminals
  5. Optionally remove the CICS.MEMBER.OUT node and save your work
  6. Stop/undeploy CICSMembertoDBStore application and original memberDBStore from the runtime
  7. Redeploy the new version of the memberDBStore RestAPI

The updated Rest API and Mapping are shown below.

runnomq2

runnomq1

Back to tutorial list | Back to top

 

Footnote:

All the collateral (end state) and documentation for the tutorials in this article is available from GitHub https://github.com/DAVEXACOM/IBMHybridIntegrationDemo

What next?

Explore other tutorials in Hybrid Integration Part 2 – IBM Application Integration Suite in a representative hybrid cloud landscape, which explores how we can move the integration platform itself from IBM Integration Bus on premises to IBM Integration Bus on Cloud – a managed service for IIB available from the IBM Marketplace.

7 comments on"Hybrid Integration Part 1 – On premises to multiple clouds. Going beyond the “Whizz-Bang” tool demos"

  1. 3. Could you please elabrate on integration b/w IIB with APP Connect
    4. Do we need to use Secure Gateway generated IP and port for common to connect like a) Bluemix –>|SG|–>IIB
    b) IIB –>|SG|–> APPConnect.

    • Hi.
      The IIB / App Connect integration is Webhook based, so kind of pub/sub over HTTP rather than messaging, with IIB acting as the pub/sub “broker” – Take a look at Webhooks in general.

      I used Secure Gateway service on Bluemix and Sec G/W client because I wanted to demonstrate Bluemix PaaS primarily. In general you wouldn’t expose IIB services (Rest or Web Services) directly to the public domain without a secure proxy of some kind at the edge. Secure Gateway is simple and convenient but there are other approaches out there. Secure Gateway is not a pre-req to solving these kind of problems but it is simple , convenient and secure. Cheers

  2. Hi Folks,
    I’m in Demo 7. So far everything is fine and I’m stuck on the script shown in this demo on docker. Could you pls tell us where can we get the script

    • Hi Amar,
      Thanks for the email. Great to hear of your progress.
      The script used in the demo is part of the collateral for the demo series (on Github). For a link to the collateral, see the article section Want to run it yourself?. Specifically, the script is in the Secure Gateway collateral.
      Regards, Ian

      • Hi Lan,
        Thanks for your quick reply. Its really helpful for me. I’ll be much happy if you provide your inputs on below things as well.

        1. Could you please provide pattern link to download. By default its not available in the toolkit with the name “IBM App Connect Custome App” as shown in demo 9/10
        2. In Demo 9 the same IP & port number was used in yaml which we got it from Secure gateway. For what purpose the same ip & port was used in swagger while creating application in AppConnect

        Thanks & Regards,
        T. Amar
        amar.it1226@gmail.com

  3. TUYA_Aleksandr_Nartovich October 26, 2016

    Thanks Dave! Great set of materials!

Join The Discussion

Your email address will not be published. Required fields are marked *