We’re giving away 1,500 DJI Tello drones. Enter to win ›
By Prabhakar Mynampati | Published May 9, 2017 - Updated May 8, 2017
API ManagementIoTPlatform as a ServiceCloudHybrid CloudOn Premises
The face of integration – your enterprise data, applications, and services – changes frequently as your business needs grow. You must adapt to new business models to sustain competition with your market peers in today’s digital world. Your requirements can also change for the following reasons:
Variances in enterprise systems pose numerous challenges to system integrators to augment existing well-established integration platforms in an organization. These variances pave the way for new integration platforms and tools, such as integration platform as a service (iPaaS), integration software as a service (iSaaS), and application programming interface (API) management suites to complement existing integration platforms. This combination of the existing integration platform with new integration suites is called the hybrid integration platform.
The challenge for solution directors and integration architects now is to build an optimized solution design for this hybrid integration platform. They must meet the technical and business service-level agreements (SLAs) under budget constraints and align with the enterprise digital transformation journey.
This article identifies three core use cases to position the hybrid integration platform and explains the implementation concepts of hybrid integration. Depending on your enterprise digital transformation strategy, you must define an appropriate hybrid integration architecture for your solution design. Based on the outcome of a suitable hybrid architecture, you can then choose a single-vendor-built platform or a build-it-yourself (BIY) platform with customization based on the needs of your enterprise.
The adaptation of enterprises to the cloud has drastically reduced the cost of enterprise technology. For reasons of agility, elasticity, and innovation, enterprises started to migrate their existing applications, data warehouses, business process management (BPM) and master data management (MDM) platforms, and mobile applications to cloud. For many enterprises, although the vision is entirely a cloud-first, cloud-ready, or cloud-only technology stack, the reality is still to mix the existing on-premises stack with newer cloud applications and services. As a result, today, application, data, and process integrations must be able to support both cloud-based applications and on-premises systems. The following diagram depicts the current landscape of systems in an enterprise and their style of integration.
The left side of this diagram illustrates the lift-and-shift of applications to the cloud without the complexity of integrations to the on-premises integration platform, but with the shift to SaaS enterprise applications. Enterprises adopt this strategy first to reduce costs.
The middle part of this diagram illustrates the next step in reducing expenses without redesign. Enterprises migrate expensive physical infrastructures of data and data warehouse workloads to the public cloud. They phase out some of the obsolete data marts that are on-premises.
The right side of the diagram illustrates how new business models are compelled by mobile technology to connect customer-facing application functions. This mobile technology must also interact with the near real-time operational data stores of an enterprise with the right APIs to expose the mobile applications.
The applications are scattered around the public and private cloud environment and within on-premises data center servers. The applications and data services are integrated through traditional integration platforms, such as enterprise service buses (ESBs); extract, transform, and load (ETL); and messaging middleware platforms. Similarly, on the cloud side, the migrated applications and the third-party applications are hosted on their own clouds (software as a service, or SaaS). They are integrated through iPaaS tools to meet some of the application use-case functions. Now, the need is to integrate on-premises applications directly or through an existing on-premises enterprise application integration (EAI) layer with cloud-migrated applications or SaaS applications to meet some of the end-to-end business process functions. You can meet this requirement by realigning the iPaaS platform or traditional on-premises EAI platform or by combining both platforms, which is called a hybrid integration platform.
On the data front, you need to synchronize or move data from the on-premises operational data stores to the database as a service (DBaaS), or data warehouse as a service (DWHaaS), on the cloud side or vice versa. You can synchronize this data by integrating both the on-premises and cloud data sources by using a managed data integration platform that is hosted on the cloud or by using the on-premises integration platforms.
Previously, mobile applications were built by connecting them on top of client-facing applications, such as SaaS applications. Now, after iPaaS platforms are deployed on the cloud, the required APIs are created from the SaaS applications, and the enterprise data is consumed from the SaaS applications or directly from the on-premises data stores through the on-premises integration. These three types of applications, data, and mobile integrations are highly demanding for a hybrid integration platform.
To generate more revenue, organizations are entering into new digital business by exposing their functions and data through APIs to the external stakeholders as use and pay services. In the past, these APIs were locked behind enterprise firewalls for internal use. These APIs are domain-specific, reusable business functions or composable interactive APIs to the front-end systems, such as web, mobile, and Internet of Things (IoT) channels. These digital APIs are implemented in two ways as explained in the following sections.
You can implement digital APIs by exposing the application data to the customer, by building composable APIs as shown in the following diagram. The existing application architectures are not changed. However, their systems of record (SOR) are extracted through SOAP or RESTful services from traditional EAI platforms. One or more system APIs are composed together based on the front-end GUI needs for presenting to a client. In this figure, for the front-end APIs, the data of one or more system APIs is composed and presented to the front-end device. For the system APIs, the core systems data is accessed and assembled in RESTful API JSON or XML objects.
Implementing APIs by using microservices is another initiative that is currently transforming existing systems to digital platforms by suitably exposing APIs for the ease of front-end channel integration. Organizations in various industries, such as banking, insurance, telecom, and travel, have successfully moved part or all of their operations to online, self-service models. To promote their business in a more intuitive way, these organizations have added new digital functions and features, such as location navigation, spot payment, and rating services. These new requirements have forced the IT domain to respond with the rapid development of new applications. The applications are built with microservices that consist of in-house IT applications along with partner systems that are deployed worldwide.
Microservices are a new way to re-design complex applications. They are built with DevOps tools and run on on-premises application servers or in Docker containers that are hosted in the cloud. These complex applications are broken down into self-contained microservice components. The function of these components is to focus on doing one specific action at a time. An independent, self-contained microservice requires its own copy of data to work with, preferably NoSQL databases. This data must be synchronized with operational databases for enterprise systems. The synchronized data is implemented by using traditional data integration patterns, such as data sync and event streams, as shown in the following diagram for the back-end systems integration. These microservices are integrated into the front-end channels and partner SaaS applications by exposing APIs that use recent API gateway technologies.
Data that is generated from IoT devices must be integrated with enterprise applications to make real-time decisions. Because some applications are on the cloud and some are on-premises, a hybrid integration platform is necessary to connect the IoT integration gateway. An IoT platform has an integration gateway that interconnects all devices by using various IoT standard channels, such as the following examples:
The IoT integration filters and aggregates the data streams onsite and sends relevant information, such as error and alert messages, to the central integration platform. The following diagram illustrates a typical IoT integration.
When you introduce a hybrid integration platform, you must have a clear strategy to position it within your organization. You must know the right place to position it to ensure optimum integration of the platform’s functional components and to adopt proper governance. To successfully implement the hybrid integration platform, you must follow these steps:
Many components constitute a hybrid integration platform because the scope of integration is across the enterprise.
An API management suite provides a complete set of tools and servers to develop and run your APIs. API management is a process of creating and publishing APIs, enforcing their usage policies, controlling access, nurturing the subscriber community, and collecting and analyzing usage statistics.
The main component of this suite is the API gateway server. This server acts as a front end, receives API requests, enforces throttling and security policies, passes calls to the backend service, and then passes the response back to caller. The gateways have some transformation engine capabilities and can orchestrate the request and responses. They also collect analytics data and provide caching, among other functions.
The other components of the API management suite are API publishing tools. These tools define and manage the API lifecycle, the development portals for the subscription of APIs by community users, reporting and analytics tools for API usage monitoring, and a monetization component to charge money for commercial APIs.
The on-premises application integration suite is a traditional vendor-specific application integration suite. It’s an ESB suite that consists of a set of tools to develop integration flows, a runtime server to run integration flows, and set of administration tools to monitor and debug mediation modules. This suite is meant for the application integration of data mapping, protocols, method call transformations, and request routing between provider and consumer applications.
Data integration tools are meant for data stores that are on-premises or on the cloud. Such techniques as Change Data Capture (CDC), ETL, and data virtualization are used for independent data structures to work together. Various vendors follow their own techniques for combining multiple sources of data or synchronizing the data from one entity to other.
B2B gateways (or managed file transfer (MFT) software) integrate data from back-end systems that enable an exchange across trading partners. B2B gateways also provide a centralized point for the transformation of multiple data sources through interoperable standards, such as XML, commerce XML (cXML), and electronic data interchange (EDI). This platform is often a component of a service-oriented architecture (SOA) in an enterprise. Other capabilities of a B2B gateway are trading partner management and security control among trading partners.
iPaaS platforms, which are a cloud version of an integration layer, are used exclusively for cloud applications and cloud data sources integration. Because iPaaS is offered as a cloud service, it is hosted only in a cloud environment. Integrations in iPaaS are much more simplified, unlike traditional ESB platforms. Therefore, ad hoc integrators manage to implement solutions with this suite. Many vendors often provide API management suite capabilities in this platform.
iSaaS platforms are cloud-delivered, prepackaged cloud streams that connect different SaaS applications as cloud-to-ground endpoints. iSaaS offers an easy-to-use interface to allow business users (called citizen integrators) to perform simple integration tasks themselves, such as integrating their Twitter feeds with their Google documents.
An IoT integration gateway is a software suite or a PaaS cloud offering. It monitors, and might even manage and control, various types of endpoints, often by using applications that users build on the platform. It facilitates operations that involve IoT endpoints and integration with enterprise resources.
This platform is constructed with different domains of integration technologies. As a result, many product vendors of existing integration technology incorporate the other areas of integration features into their own products and position them in the market.
In terms of security and performance, many challenges and risks are involved in designing a hybrid integration platform while you maintain your existing integration infrastructure. The deployment of newly developed mediation services and migrated mediation services must not disrupt business continuity. For a smooth transition and deployment in production, before you customize the platform to deploy into production, build and test a master proof-of-concept (PoC) of your hybrid integration platform.
To optimize the design of your hybrid integration platform:
The following table outlines the most suitable integration platforms based on the type of integration and the deployment environment in which the integration occurs.
The following table highlights the most suitable integration platforms based on the type of integration and its endpoint connectivity to the target system deployment environment.
To help you understand the design philosophy, consider this example. A company engages an integration architect who conducts a study on the company’s as-is integration architecture. The enterprise design team explains to the integration architect the strategy for transforming the application landscape over the next 3 to 5 years. Based on the findings of the company’s architecture, the integration architect begins with the details that are outlined in Table 3 for the future integration landscape.
As explained in Step 4, the existing integration platforms are replaced with the suggested reference integration platforms. The following figure shows the initial version of the hybrid integration platform based on the information in Table 2 and Table 3.
As explained in step 5, the hybrid integration platform is optimized with the following considerations:
The following figure illustrates the optimized hybrid integration platform.
This article highlighted three core use cases to position the hybrid integration platform and the implementation concepts of hybrid integration. You learned how a single integration platform is insufficient to meet all the integration requirements of an enterprise because of the various deployment environments, operating models, and new technologies that are added to the existing setup. Fortunately, a single-vendor product stack for building a hybrid integration platform meets the standards of interoperability and avoids redundant integration of functional components among the various types of integration products.
I thank Kim Clark, Integration Portfolio Architect for IBM UK, for taking the time review and provide feedback on the article.
Back to top