By Kim Clark, Rob Nicholson | Updated June 23, 2016 - Published June 22, 2016
API ManagementMessagingPlatform as a ServiceCloudHybrid Cloud
Over time, integration inevitably increases in complexity. This complexity is a result of the greater diversity of resources that we need to integrate, in ever-increasing permutations of infrastructures and platforms. Furthermore, the people who are involved in integrating systems are no longer centralized in a single technical team, but are spread throughout and beyond the enterprise.
In parallel, and in part as a result of this increase in complexity, a competing drive aims to simplify and rationalize integration. Web APIs have matured to become a common platform and language-agnostic way for applications to communicate. Infrastructure is ever more virtualized and containerized to free run times from hardware and operating system specifics and enable elastic workload orchestration. Teams are learning to more effectively join their development and operations staff together and automate from build to deployment for rapid release cycles.
The challenges are truly hybrid in many different senses of the word. Locations, teams, methods, platforms, and more are continuously diverging. The architecture of the integration across this hybrid environment is evolving at a rapid pace.
This article explores how hybrid integration has evolved. First, we examine how the scope of integration has changed with the move to cloud. Next, we define the high-level characteristics of a hybrid integration capability. Then, we explain how IT is less often a central function within the organization. We continue by looking at the fundamental building blocks of a hybrid integration architecture and how integration can be productized though the API economy. Finally, we highlight how you can recognize and satisfy the needs of the digital team and improve consistency across the hybrid environment.
Today, the ownership boundary of an enterprise spreads well beyond its walls, encompassing IT assets across a truly hybrid environment. In this architecture, existing applications are moved to the infrastructure as a service (IaaS) of cloud providers. New applications are often built on the cloud as a platform as a service (PaaS). In line with this trend, a surge is also taking place in the use of pre-built cloud-based software as a service (SaaS).
In addition, interaction with external parties, whether customers or business partners, is increasingly automated. The degree of automation with those external parties is often a business differentiator.
As such, any integration capability must fundamentally address connectivity across cloud boundaries. This capability must also simplify the security and management issues that it brings and embrace the standards that are evolving within hybrid architectures.
Hybrid integration is often simplistically defined as the ability to integrate on-premises systems with cloud systems. The reality for most organizations has much broader dimensions. A true hybrid integration architecture considers integration between all the owned environments, spanning on-premises and cloud environments, and whether that cloud is local, dedicated, or public. It also spans from a self-built environment to platforms to SaaS. It also must factor how the enterprise connects with its partners and customers. Hybrid integration has a vast scope. The key challenge is how to interpret that complexity and find common architectural patterns within it to help simplify the problem.
At a high level, hybrid integration is a broad integration framework. It seamlessly bridges all owned environments, whether direct data sources, applications, or APIs, and can connect to them wherever they might be: on-premises, IaaS, PaaS, or SaaS.
Hybrid integration must contain broad connectivity capabilities for modern cloud-based applications and to equally critical, but older systems of record (SOR). It must have tools to simplify and accelerate productivity, such as flexible and fast mapping and transformation. From a non-functional perspective, it must also provide enterprise-grade and Internet-grade scalability, security, and resilience.
However, although it is important to define hybrid integration as a whole, we must consider how integration needs are changing and recognize the two different audiences that are involved.
For some years now, particularly accelerated by the mobile revolution, centralized IT has diverged into at least two different camps: often termed enterprise IT and digital IT.
Enterprise IT retains its vital role of ensuring that business-critical systems maintain their service levels and provide the type of data integrity and secure governance that are expected of core systems on which the business depends. However, this heavily regulated and controlled environment does not meet the needs of the lines of business (LOBs) who must keep the public face of the enterprise fresh with new propositions and innovations in a rapidly changing market.
LOBs have an increasing focus on requirements, such as the following examples:
Given these requirements, you can see how shadow IT departments have arisen within LOBs and are now taking on significant new projects by using techniques that are different from the techniques that are used by central IT. These LOB IT departments are quickly moving out of the shadows and becoming recognized as the digital IT team who is responsible for creating the next generation of applications. This team often has a different culture and focus, recruiting business minded people with technical skills rather than raw technical specialists. As such, the team’s focus is on gaining revenue and market share by using rapid innovation. However, the applications that these individuals are creating represent the public face of the company. Therefore, they also require a solid technical backbone to satisfy core non-functional requirements.
Both enterprise IT and digital IT teams have integration needs. They need to integrate both within their own domains and with one another. At a high level, those hybrid integration capabilities sound similar, such as connectivity, transformation, API exposure, and composition. However, they differ dramatically in their details.
In a simplistic reference architecture for integration, such as the example shown in the following diagram, a central IT team might perform most of the integration. The team essentially bridges the bi-modal divide by empowering the digital teams. It performs the deep integration that is needed to surface the systems of record as APIs and events on the central API gateway.
Another important but subtle aspect of the diagram is cloud affinity. This concept deliberately represents a continuum rather than a dividing line on the architecture. As explained for the surface area of hybrid integration, any part of the architecture can be on-premises or fully cloud-based. The components near the bottom of the diagram, such as systems of record, are more likely to be on-premises, but a new system of record might be deployed on a cloud infrastructure, or even be SaaS. Similarly, components near the top of the diagram are more likely to be cloud-based, but plenty of organizations still host their own systems of engagement applications. The reality for any organization is a blend of on-premises and off-premises components, and any hybrid integration architecture must enable connectivity regardless.
For this core enterprise integration, the requirements are broad and well known, with the following primary concerns:
Later in this article, we contrast these requirements with the requirements of digital IT, but first we consider how they relate to typical integration initiatives, such as service-oriented architecture (SOA).
Some might equate the bi-modal integration diagram to SOA, but with more modern protocols for service or API exposure. In fact, the evolution from early SOA is more pronounced. The sophistication and purpose of the gateway has matured significantly. Modern API Gateways have a more consumer-focused experience. They provide developer portals to browse and subscribe to APIs, analytics on API usage, and modern security models and traffic management for safe and controlled access to APIs.
The evolved richness in mechanisms for provisioning of APIs has led companies to explore public exposure of APIs and to join the API economy. Architecturally, you can recognize this exposure by a second (upper) gateway in the architecture as shown in the following diagram. This section looks at how and why the public gateway differs from the internal gateway.
Although we show two separate gateways in this logical reference architecture to emphasize the two styles of exposure, these gateways might be provided by the same physical gateway in some scenarios.
The concept of systems of engagement generally refers to user-facing applications, such as mobile apps and single page web apps. These applications address modern digital channels with responsive, high-traction user interfaces that drive new revenue opportunities for the business. However, increasingly a new business channel can be introduced in the form of publicly accessible APIs. These APIs are built with external developers in mind, aiming to accelerate innovation by crowd sourcing new business models. Although technically similar to the internal APIs, public APIs are radically different in how, why, and where they are created, and who creates them:
To technically enable public API exposure, broader capabilities are required within and around the gateway. For example, public APIs might have more advanced and robust subscription-based traffic management. They might have digital portals through which developers can discover and subscribe to APIs. They might also have Internet security models (such as OAuth), deeper threat protection, more marketing focused analytics, and a direct developer feedback mechanism, such as community support forums.
For digital IT teams to provide new, engaging, and coherent APIs, they need to do more than simply re-expose internal APIs. They need to compose existing APIs within and beyond the company to provide capabilities that can precisely address new market channels. They have much less focus on the low-level integration problems of complex protocols and diverse platforms that enterprise IT deal with. The applications and APIs that these teams create communicate primarily by using more recently popularized protocols, such as REST (typically JSON/HTTP) with little integration heavy lifting. Therefore, they can focus more on adding functionality.
A successful modern mobile application serves vast numbers of concurrently active users, each demanding a subsecond response time. This demand drives a need to bring data closer to the edge of the enterprise rather than retrieving it at run time from systems of record that are not designed with low latency and mass market scalability in mind. The result is the need for data movement patterns, caching, and potentially more complex persistence patterns, such as eventual consistency.
Digital teams regularly use lightweight asynchronous communication internally to reduce direct dependencies and provide greater resilience and scalability. They also look to more asynchronous methods externally, for example by using a push versus pull model for mobile applications. Digital teams often turn to a microservices architecture to help them bring these technology patterns to the forefront and with the goal of improving agility, scalability, and resilience. Microservices and their relationship to integration are too complex to describe in this article. For a more detailed explanation, see Microservices, SOA, and APIs: Friends or enemies?.
Based on the following figure, when you look more deeply into the integration needs of digital teams, you see a need for something more than just a gateway to expose their APIs. Although these new applications and APIs can be crafted by using raw language run times, many of the composition challenges are well-known integration patterns, such as mapping and enrichment. Therefore, it makes sense to use simple tools and frameworks to accelerate the implementation of those APIs.
In addition to the external gateway for APIs, digital teams also have the following integration needs:
A hybrid integration architecture must enable frictionless integration across the enterprise, with the following qualities:
Conceptually, integration should feel similar for everyone, even though your specific needs might vary. To achieve these competing demands, you must look for ways to simplify the problem.
As an example, a common way to achieve consistency in an architecture is to look for repeating patterns and use them to simplify the architecture. In the architecture that is presented in this article, a repeating pattern is clearly visible across the hybrid integration space and consists of two core elements:
These two elements are repeated in varying degrees across the integration landscape, although the depth and sophistication that are required from each of these two elements varies by scenario.
For example, in the lower layers that are shown in the following diagram, when you connect deeply to existing systems of record, you are likely to require the full toolbox of capabilities. Now, compare this arrangement to the upper layers of the diagram. Here, digital teams that expose new APIs based on the composition of existing internal APIs, and external partners, might require only the gateway and some of the more lightweight composition capabilities. Teams that create new applications might need little more than a gateway to expose their APIs and access the already exposed internal APIs.
This view of the architecture provides a look into a much broader set of opportunities for simplification. By targeting elements of consistency, such as the ones shown here, we encourage a common, but flexible pattern to perform integration. We simplify and rationalize the architecture without sacrificing capability. We aim to explore these areas of consistency with flexibility further in future articles.
This article explored the evolution of hybrid integration. It explained how hybrid integration has extended the ownership boundaries of an enterprise. It defined the core capabilities of hybrid integration and the role and requirements of the lines of business. Then, it demonstrated how the central IT team bridges the bi-modal divide, empowering the digital team. It differentiated between public APIs and internal APIs. Finally, this article demonstrated the importance of meeting the needs of the digital team and ensuring consistency with flexibility in the hybrid environment.
In this article, we deliberately avoid making references to IBM products, but as you can imagine, we have a strong point of view on them. Through the remainder of 2016, watch for posts on the IBM Integration Developer Center blog that map the reference architecture to IBM offerings.
Thank you to the following people for their input and review of the material in this article: Andy Garratt, Peter Broadhurst, Carsten Bornert, Guy Hochstetler, and Carlo Marcoli.
Back to top