Hybrid integration platform in the digital enterprise
Use cases and concepts to help you define an appropriate hybrid integration architecture
The face of integration — your enterprise data, applications, and services — changes frequently as your business needs grow. You must adapt to new business models to sustain competition with your market peers in today’s digital world. Your requirements can also change for the following reasons:
- The addition of applications that are built with new technologies
- The migration of existing applications
- The introduction of third-party systems and services to various types of cloud-hosting services that are available in the market
- The addition of new business channels, partners, suppliers, and other business units and organizations
- The integration of these enterprise applications with emerging pervasive computing devices and sensors to ensure that enterprise systems support the real-time or near real-time needs of the business
Variances in enterprise systems pose numerous challenges to system integrators to augment existing well-established integration platforms in an organization. These variances pave the way for new integration platforms and tools, such as integration platform as a service (iPaaS), integration software as a service (iSaaS), and application programming interface (API) management suites to complement existing integration platforms. This combination of the existing integration platform with new integration suites is called the hybrid integration platform.
The challenge for solution directors and integration architects now is to build an optimized solution design for this hybrid integration platform. They must meet the technical and business service-level agreements (SLAs) under budget constraints and align with the enterprise digital transformation journey.
This article identifies three core use cases to position the hybrid integration platform and explains the implementation concepts of hybrid integration. Depending on your enterprise digital transformation strategy, you must define an appropriate hybrid integration architecture for your solution design. Based on the outcome of a suitable hybrid architecture, you can then choose a single-vendor-built platform or a build-it-yourself (BIY) platform with customization based on the needs of your enterprise.
Use case A: Demand for hosting apps, data, and services on the cloud
The adaptation of enterprises to the cloud has drastically reduced the cost of enterprise technology. For reasons of agility, elasticity, and innovation, enterprises started to migrate their existing applications, data warehouses, business process management (BPM) and master data management (MDM) platforms, and mobile applications to cloud. For many enterprises, although the vision is entirely a cloud-first, cloud-ready, or cloud-only technology stack, the reality is still to mix the existing on-premises stack with newer cloud applications and services. As a result, today, application, data, and process integrations must be able to support both cloud-based applications and on-premises systems. The following diagram depicts the current landscape of systems in an enterprise and their style of integration.
The left side of this diagram illustrates the lift-and-shift of applications to the cloud without the complexity of integrations to the on-premises integration platform, but with the shift to SaaS enterprise applications. Enterprises adopt this strategy first to reduce costs.
The middle part of this diagram illustrates the next step in reducing expenses without redesign. Enterprises migrate expensive physical infrastructures of data and data warehouse workloads to the public cloud. They phase out some of the obsolete data marts that are on-premises.
The right side of the diagram illustrates how new business models are compelled by mobile technology to connect customer-facing application functions. This mobile technology must also interact with the near real-time operational data stores of an enterprise with the right APIs to expose the mobile applications.
The applications are scattered around the public and private cloud environment and within on-premises data center servers. The applications and data services are integrated through traditional integration platforms, such as enterprise service buses (ESBs); extract, transform, and load (ETL); and messaging middleware platforms. Similarly, on the cloud side, the migrated applications and the third-party applications are hosted on their own clouds (software as a service, or SaaS). They are integrated through iPaaS tools to meet some of the application use-case functions. Now, the need is to integrate on-premises applications directly or through an existing on-premises enterprise application integration (EAI) layer with cloud-migrated applications or SaaS applications to meet some of the end-to-end business process functions. You can meet this requirement by realigning the iPaaS platform or traditional on-premises EAI platform or by combining both platforms, which is called a hybrid integration platform.
On the data front, you need to synchronize or move data from the on-premises operational data stores to the database as a service (DBaaS), or data warehouse as a service (DWHaaS), on the cloud side or vice versa. You can synchronize this data by integrating both the on-premises and cloud data sources by using a managed data integration platform that is hosted on the cloud or by using the on-premises integration platforms.
Previously, mobile applications were built by connecting them on top of client-facing applications, such as SaaS applications. Now, after iPaaS platforms are deployed on the cloud, the required APIs are created from the SaaS applications, and the enterprise data is consumed from the SaaS applications or directly from the on-premises data stores through the on-premises integration. These three types of applications, data, and mobile integrations are highly demanding for a hybrid integration platform.
Use case B: Adapting the digital API economy
To generate more revenue, organizations are entering into new digital business by exposing their functions and data through APIs to the external stakeholders as use and pay services. In the past, these APIs were locked behind enterprise firewalls for internal use. These APIs are domain-specific, reusable business functions or composable interactive APIs to the front-end systems, such as web, mobile, and Internet of Things (IoT) channels. These digital APIs are implemented in two ways as explained in the following sections.
Expose a system of record through interactive APIs
You can implement digital APIs by exposing the application data to the customer, by building composable APIs as shown in the following diagram. The existing application architectures are not changed. However, their systems of record (SOR) are extracted through SOAP or RESTful services from traditional EAI platforms. One or more system APIs are composed together based on the front-end GUI needs for presenting to a client. In this figure, for the front-end APIs, the data of one or more system APIs is composed and presented to the front-end device. For the system APIs, the core systems data is accessed and assembled in RESTful API JSON or XML objects.
Implement APIs by using microservices principles
Implementing APIs by using microservices is another initiative that is currently transforming existing systems to digital platforms by suitably exposing APIs for the ease of front-end channel integration. Organizations in various industries, such as banking, insurance, telecom, and travel, have successfully moved part or all of their operations to online, self-service models. To promote their business in a more intuitive way, these organizations have added new digital functions and features, such as location navigation, spot payment, and rating services. These new requirements have forced the IT domain to respond with the rapid development of new applications. The applications are built with microservices that consist of in-house IT applications along with partner systems that are deployed worldwide.
Microservices are a new way to re-design complex applications. They are built with DevOps tools and run on on-premises application servers or in Docker containers that are hosted in the cloud. These complex applications are broken down into self-contained microservice components. The function of these components is to focus on doing one specific action at a time. An independent, self-contained microservice requires its own copy of data to work with, preferably NoSQL databases. This data must be synchronized with operational databases for enterprise systems. The synchronized data is implemented by using traditional data integration patterns, such as data sync and event streams, as shown in the following diagram for the back-end systems integration. These microservices are integrated into the front-end channels and partner SaaS applications by exposing APIs that use recent API gateway technologies.
Use case C: Extending the enterprise for IoT
Data that is generated from IoT devices must be integrated with enterprise applications to make real-time decisions. Because some applications are on the cloud and some are on-premises, a hybrid integration platform is necessary to connect the IoT integration gateway. An IoT platform has an integration gateway that interconnects all devices by using various IoT standard channels, such as the following examples:
- MQ Telemetry Transport (MQTT)
- Constrained Application Protocol (CoaP)
- Near Field Communication (NFC)
- Radio-frequency identification (RFID)
- Universal Serial Bus (USB)
- Serial Peripheral Interface (SPI) Bus
- Inter-Integrated Circuit (I2C)
The IoT integration filters and aggregates the data streams onsite and sends relevant information, such as error and alert messages, to the central integration platform. The following diagram illustrates a typical IoT integration.
Creating a strategy for hybrid integration platform
When you introduce a hybrid integration platform, you must have a clear strategy to position it within your organization. You must know the right place to position it to ensure optimum integration of the platform’s functional components and to adopt proper governance. To successfully implement the hybrid integration platform, you must follow these steps:
- Plan the location. Plan where to host the infrastructure in the organization. Locate the “center of gravity” of the application landscape. Consider your three-to-five year plan of application transformation strategy, and then determine whether it points to the cloud or on-premises. The closer this platform is to the applications and data, the better it is to maintain quality-of-service (QoS) requirements. You can choose to host your integration on-premises, in the cloud, or both (hybrid cloud).
- Determine ownership. Decide who will control and own on this platform and the IT applications. Because you have more flexibility in the control of IT systems based on cloud service offerings today, choose whether private/public cloud integration, hybrid cloud integration, or managed service cloud integration is the best fit for your purposes. These variants provide full, partial, and complete outsourced control to the platform.
- Identify resources. Identify the skilled resources who will implement the platform. The on-premises application integration demands highly skilled integration specialists that are knowledgeable about the vendor platform. For application integration on the cloud, developers are sufficient because the iPaaS platform is equipped with the necessary tools. For iSaaS platforms, the integration is implemented by business analysts, called citizen integrators, because they work with the platform GUI to map the systems.
- Ensure governance and control. When new integration options and approaches are introduced, they can impact or corrupt the data quality of your systems of record. For example, a developer might not have right level of knowledge about the system of record and inadvertently corrupt it. Ensure that only authorized users have access to the application data, and that you have a proper level of governance and controls in place to mitigate risks.
Architectural components of a hybrid integration platform
Many components constitute a hybrid integration platform because the scope of integration is across the enterprise.
API management suite
An API management suite provides a complete set of tools and servers to develop and run your APIs. API management is a process of creating and publishing APIs, enforcing their usage policies, controlling access, nurturing the subscriber community, and collecting and analyzing usage statistics.
The main component of this suite is the API gateway server. This server acts as a front end, receives API requests, enforces throttling and security policies, passes calls to the backend service, and then passes the response back to caller. The gateways have some transformation engine capabilities and can orchestrate the request and responses. They also collect analytics data and provide caching, among other functions.
The other components of the API management suite are API publishing tools. These tools define and manage the API lifecycle, the development portals for the subscription of APIs by community users, reporting and analytics tools for API usage monitoring, and a monetization component to charge money for commercial APIs.
On-premises application integration suite
The on-premises application integration suite is a traditional vendor-specific application integration suite. It’s an ESB suite that consists of a set of tools to develop integration flows, a runtime server to run integration flows, and set of administration tools to monitor and debug mediation modules. This suite is meant for the application integration of data mapping, protocols, method call transformations, and request routing between provider and consumer applications.
Data integration tools
Data integration tools are meant for data stores that are on-premises or on the cloud. Such techniques as Change Data Capture (CDC), ETL, and data virtualization are used for independent data structures to work together. Various vendors follow their own techniques for combining multiple sources of data or synchronizing the data from one entity to other.
B2B integration software
B2B gateways (or managed file transfer (MFT) software) integrate data from back-end systems that enable an exchange across trading partners. B2B gateways also provide a centralized point for the transformation of multiple data sources through interoperable standards, such as XML, commerce XML (cXML), and electronic data interchange (EDI). This platform is often a component of a service-oriented architecture (SOA) in an enterprise. Other capabilities of a B2B gateway are trading partner management and security control among trading partners.
Integration platform as a service
iPaaS platforms, which are a cloud version of an integration layer, are used exclusively for cloud applications and cloud data sources integration. Because iPaaS is offered as a cloud service, it is hosted only in a cloud environment. Integrations in iPaaS are much more simplified, unlike traditional ESB platforms. Therefore, ad hoc integrators manage to implement solutions with this suite. Many vendors often provide API management suite capabilities in this platform.
Integration software as a service
iSaaS platforms are cloud-delivered, prepackaged cloud streams that connect different SaaS applications as cloud-to-ground endpoints. iSaaS offers an easy-to-use interface to allow business users (called citizen integrators) to perform simple integration tasks themselves, such as integrating their Twitter feeds with their Google documents.
IoT integration gateways
An IoT integration gateway is a software suite or a PaaS cloud offering. It monitors, and might even manage and control, various types of endpoints, often by using applications that users build on the platform. It facilitates operations that involve IoT endpoints and integration with enterprise resources.
This platform is constructed with different domains of integration technologies. As a result, many product vendors of existing integration technology incorporate the other areas of integration features into their own products and position them in the market.
Design method of a hybrid integration platform
In terms of security and performance, many challenges and risks are involved in designing a hybrid integration platform while you maintain your existing integration infrastructure. The deployment of newly developed mediation services and migrated mediation services must not disrupt business continuity. For a smooth transition and deployment in production, before you customize the platform to deploy into production, build and test a master proof-of-concept (PoC) of your hybrid integration platform.
To optimize the design of your hybrid integration platform:
- Identify the type of integration. Also, determine the number of integration touch points (that is, on-premises, IaaS, or PaaS of an organization) within each deployment.
- Identify the integration endpoints from one deployment environment to other. Among these end-point types, an enterprise can have on-premises to cloud, cloud to on-premises, SaaS to IaaS applications, SaaS to on-premises, enterprise cloud to third-party cloud, and on-premises to on-premises.
- Identify the existing integration middleware and techniques that are used. For example, consider whether you are using a point–to–point messaging system or file transfer, integration servers (such as brokers), a service bus, or gateways.
- Replace the existing integration platforms with the suggested reference integration platforms that are mentioned in Table 1 and Table 2.
- Optimize or consolidate the hybrid integration platform from the initial design of standard integration platforms.
- Finalize the suitability of deployment of the hybrid integration platform as close as to the center of gravity of applications and data stores.
The following table outlines the most suitable integration platforms based on the type of integration and the deployment environment in which the integration occurs.
Table 1. A reference for standard integration platforms
|Data integration||ETL||Cloud ETL||iPaaS|
|SOA services||ESB||Cloud ESB||iPaaS|
|Process integration||BPEL||Cloud BPEL||PaaS BPEL|
|Microservices||API gateway||API gateway||API gateway|
|IoT channels||IoTgateway||IoT gateway||IoT gateway|
The following table highlights the most suitable integration platforms based on the type of integration and its endpoint connectivity to the target system deployment environment.
Table 2. A reference for standard platforms for integration endpoints
|Integration type||On premises to on-premises (different enterprises)||On premises to cloud or vice versa|
|Data integration||Vendor-specific tools||iPaaS|
|SOA services||ESB gateway||iPaaS|
|IoT channels||IoT gateway||IoT gateway|
|SaaS apps||Not Applicable||iPaaS|
To help you understand the design philosophy, consider this example. A company engages an integration architect who conducts a study on the company’s as-is integration architecture. The enterprise design team explains to the integration architect the strategy for transforming the application landscape over the next 3 to 5 years. Based on the findings of the company’s architecture, the integration architect begins with the details that are outlined in Table 3 for the future integration landscape.
Table 3: Integration details for the Acme Company
|Type of integration||Number of touch points: On-premises||Number of touch points: IaaS||Number of touch points: PaaS|
|A2A|| Large |
| Medium |
(50< size >25)
| Small |
(size <25) < td>
As explained in Step 4, the existing integration platforms are replaced with the suggested reference integration platforms. The following figure shows the initial version of the hybrid integration platform based on the information in Table 2 and Table 3.
As explained in step 5, the hybrid integration platform is optimized with the following considerations:
- Because the data integration touch points are few in both the IaaS and PaaS cloud environments, the proposed cloud version of ETL is replaced with an iPaaS solution.
- Because the B2B gateway is in both environments, the iPaaS that is deployed in the cloud handles both B2B gateway functions.
- Positioning the API gateway in the cloud can be a redundant solution because iPaaS was introduced and therefore, eliminated from the initial design.
- The API gateway that is on-premises is untouched because it was demanding for high QoS requirements from the on-premises systems integration through the ESB layer.
The following figure illustrates the optimized hybrid integration platform.
This article highlighted three core use cases to position the hybrid integration platform and the implementation concepts of hybrid integration. You learned how a single integration platform is insufficient to meet all the integration requirements of an enterprise because of the various deployment environments, operating models, and new technologies that are added to the existing setup. Fortunately, a single-vendor product stack for building a hybrid integration platform meets the standards of interoperability and avoids redundant integration of functional components among the various types of integration products.
I thank Kim Clark, Integration Portfolio Architect for IBM UK, for taking the time review and provide feedback on the article.