The evolving hybrid integration reference architecture

Over time, integration inevitably increases in complexity. This complexity is a result of the greater diversity of resources that we need to integrate, in ever-increasing permutations of infrastructures and platforms. Furthermore, the people who are involved in integrating systems are no longer centralized in a single technical team, but are spread throughout and beyond the enterprise.

In parallel, and in part as a result of this increase in complexity, a competing drive aims to simplify and rationalize integration. Web APIs have matured to become a common platform and language-agnostic way for applications to communicate. Infrastructure is ever more virtualized and containerized to free run times from hardware and operating system specifics and enable elastic workload orchestration. Teams are learning to more effectively join their development and operations staff together and automate from build to deployment for rapid release cycles.

The challenges are truly hybrid in many different senses of the word. Locations, teams, methods, platforms, and more are continuously diverging. The architecture of the integration across this hybrid environment is evolving at a rapid pace.

This article explores how hybrid integration has evolved. First, we examine how the scope of integration has changed with the move to cloud. Next, we define the high-level characteristics of a hybrid integration capability. Then, we explain how IT is less often a central function within the organization. We continue by looking at the fundamental building blocks of a hybrid integration architecture and how integration can be productized though the API economy. Finally, we highlight how you can recognize and satisfy the needs of the digital team and improve consistency across the hybrid environment.

The extended surface area of hybrid integration

Today, the ownership boundary of an enterprise spreads well beyond its walls, encompassing IT assets across a truly hybrid environment. In this architecture, existing applications are moved to the infrastructure as a service (IaaS) of cloud providers. New applications are often built on the cloud as a platform as a service (PaaS). In line with this trend, a surge is also taking place in the use of pre-built cloud-based software as a service (SaaS).

In addition, interaction with external parties, whether customers or business partners, is increasingly automated. The degree of automation with those external parties is often a business differentiator.

As such, any integration capability must fundamentally address connectivity across cloud boundaries. This capability must also simplify the security and management issues that it brings and embrace the standards that are evolving within hybrid architectures.

Hybrid integration is often simplistically defined as the ability to integrate on-premises systems with cloud systems. The reality for most organizations has much broader dimensions. A true hybrid integration architecture considers integration between all the owned environments, spanning on-premises and cloud environments, and whether that cloud is local, dedicated, or public. It also spans from a self-built environment to platforms to SaaS. It also must factor how the enterprise connects with its partners and customers. Hybrid integration has a vast scope. The key challenge is how to interpret that complexity and find common architectural patterns within it to help simplify the problem.

Hybrid integration core capabilities

At a high level, hybrid integration is a broad integration framework. It seamlessly bridges all owned environments, whether direct data sources, applications, or APIs, and can connect to them wherever they might be: on-premises, IaaS, PaaS, or SaaS.

Hybrid integration must contain broad connectivity capabilities for modern cloud-based applications and to equally critical, but older systems of record (SOR). It must have tools to simplify and accelerate productivity, such as flexible and fast mapping and transformation. From a non-functional perspective, it must also provide enterprise-grade and Internet-grade scalability, security, and resilience.

However, although it is important to define hybrid integration as a whole, we must consider how integration needs are changing and recognize the two different audiences that are involved.

Adoption of IT by the line of business

For some years now, particularly accelerated by the mobile revolution, centralized IT has diverged into at least two different camps: often termed enterprise IT and digital IT.

Enterprise IT retains its vital role of ensuring that business-critical systems maintain their service levels and provide the type of data integrity and secure governance that are expected of core systems on which the business depends. However, this heavily regulated and controlled environment does not meet the needs of the lines of business (LOBs) who must keep the public face of the enterprise fresh with new propositions and innovations in a rapidly changing market.

LOBs have an increasing focus on requirements, such as the following examples:

• Agility. LOBs need to explore and adapt, rapidly iterating over prototypes, and where necessary, failing fast and moving on to the next idea. This approach implies the use of different techniques to build more componentized applications, such as a microservices architecture. It also means increasing focus on a DevOps culture, where the distance between implementation changes and production delivery is minimized.
• Scalability. If a prototype is launched, LOBs must be able to elastically scale it, moving a prototype from a handful of sponsored users to the open market, without significant additional development effort. LOBs must be able to scale the infrastructure up and down by using a simple and predictable cost model, implying use of IaaS or PaaS, and by adhering to an inherently scalable application architecture.
• Latency. LOBs might also have different needs regarding the real-time responsiveness of the applications they provide. The latency that is required to create an engaging mobile experience might be significantly lower than the latency that is provided by existing systems of record. The latency might require using pre-aggregated or cached data and designing around the challenging issues of data duplication and eventual consistency. It might also require adequate traffic management to prioritize workloads.
• Availability. The reputation of a primarily online business can be irreversibly damaged by even the smallest amount of downtime at the wrong moment. Proven availability is a differentiator. LOBs need applications that are designed for continuous high availability. These applications often require different approaches to resilience than you might choose for internal enterprise applications.

Given these requirements, you can see how shadow IT departments have arisen within LOBs and are now taking on significant new projects by using techniques that are different from the techniques that are used by central IT. These LOB IT departments are quickly moving out of the shadows and becoming recognized as the digital IT team who is responsible for creating the next generation of applications. This team often has a different culture and focus, recruiting business minded people with technical skills rather than raw technical specialists. As such, the team's focus is on gaining revenue and market share by using rapid innovation. However, the applications that these individuals are creating represent the public face of the company. Therefore, they also require a solid technical backbone to satisfy core non-functional requirements.

Both enterprise IT and digital IT teams have integration needs. They need to integrate both within their own domains and with one another. At a high level, those hybrid integration capabilities sound similar, such as connectivity, transformation, API exposure, and composition. However, they differ dramatically in their details.

Bridging the bi-modal divide

In a simplistic reference architecture for integration, such as the example shown in the following diagram, a central IT team might perform most of the integration. The team essentially bridges the bi-modal divide by empowering the digital teams. It performs the deep integration that is needed to surface the systems of record as APIs and events on the central API gateway.

Another important but subtle aspect of the diagram is cloud affinity. This concept deliberately represents a continuum rather than a dividing line on the architecture. As explained for the surface area of hybrid integration, any part of the architecture can be on-premises or fully cloud-based. The components near the bottom of the diagram, such as systems of record, are more likely to be on-premises, but a new system of record might be deployed on a cloud infrastructure, or even be SaaS. Similarly, components near the top of the diagram are more likely to be cloud-based, but plenty of organizations still host their own systems of engagement applications. The reality for any organization is a blend of on-premises and off-premises components, and any hybrid integration architecture must enable connectivity regardless.

For this core enterprise integration, the requirements are broad and well known, with the following primary concerns:

• Low-level connectivity across a broad range of data formats, transports, and protocol to ensure engagement with any systems of record they currently own or acquire, regardless of age or platform diversity.
• Data integrity by using transactions to ensure systems of record are kept in a consistent state.
• Technical API or service exposure to a limited audience of other applications that are predominantly within the organization (including data sources from digital IT) using more modern techniques, such as RESTful APIs and web services.
• Enterprise messaging facilities to provide assured delivery of critical business data between systems on a multitude of platforms.

Later in this article, we contrast these requirements with the requirements of digital IT, but first we consider how they relate to typical integration initiatives, such as service-oriented architecture (SOA).

Some might equate the bi-modal integration diagram to SOA, but with more modern protocols for service or API exposure. In fact, the evolution from early SOA is more pronounced. The sophistication and purpose of the gateway has matured significantly. Modern API Gateways have a more consumer-focused experience. They provide developer portals to browse and subscribe to APIs, analytics on API usage, and modern security models and traffic management for safe and controlled access to APIs.

Public API exposure

The evolved richness in mechanisms for provisioning of APIs has led companies to explore public exposure of APIs and to join the API economy. Architecturally, you can recognize this exposure by a second (upper) gateway in the architecture as shown in the following diagram. This section looks at how and why the public gateway differs from the internal gateway.

Although we show two separate gateways in this logical reference architecture to emphasize the two styles of exposure, these gateways might be provided by the same physical gateway in some scenarios.

The concept of systems of engagement generally refers to user-facing applications, such as mobile apps and single page web apps. These applications address modern digital channels with responsive, high-traction user interfaces that drive new revenue opportunities for the business. However, increasingly a new business channel can be introduced in the form of publicly accessible APIs. These APIs are built with external developers in mind, aiming to accelerate innovation by crowd sourcing new business models. Although technically similar to the internal APIs, public APIs are radically different in how, why, and where they are created, and who creates them:

• How. Public APIs are built and evolve rapidly as market trends shift. As such, they are typically built on a lightweight infrastructure and on design paradigms, such as microservices, enabling greater agility and, if successful, instant elastic scalability.
• Why. Public APIs should be viewed more correctly as products for sale within the API economy by using various funding models. They are often designed for consumer niches rather than as all-encompassing generic interfaces.
• Who. Public APIs are often created and maintained by a separate digital IT team that is more business-focused or market-focused and that specializes in lightweight architectures.
• Where. Public APIs are likely to be built and deployed on the cloud to take advantage of the fast ramp-up time for new environments and linear elastic scaling cost models.

To technically enable public API exposure, broader capabilities are required within and around the gateway. For example, public APIs might have more advanced and robust subscription-based traffic management. They might have digital portals through which developers can discover and subscribe to APIs. They might also have Internet security models (such as OAuth), deeper threat protection, more marketing focused analytics, and a direct developer feedback mechanism, such as community support forums.

Integration needs within digital IT

For digital IT teams to provide new, engaging, and coherent APIs, they need to do more than simply re-expose internal APIs. They need to compose existing APIs within and beyond the company to provide capabilities that can precisely address new market channels. They have much less focus on the low-level integration problems of complex protocols and diverse platforms that enterprise IT deal with. The applications and APIs that these teams create communicate primarily by using more recently popularized protocols, such as REST (typically JSON/HTTP) with little integration heavy lifting. Therefore, they can focus more on adding functionality.

A successful modern mobile application serves vast numbers of concurrently active users, each demanding a subsecond response time. This demand drives a need to bring data closer to the edge of the enterprise rather than retrieving it at run time from systems of record that are not designed with low latency and mass market scalability in mind. The result is the need for data movement patterns, caching, and potentially more complex persistence patterns, such as eventual consistency.

Digital teams regularly use lightweight asynchronous communication internally to reduce direct dependencies and provide greater resilience and scalability. They also look to more asynchronous methods externally, for example by using a push versus pull model for mobile applications. Digital teams often turn to a microservices architecture to help them bring these technology patterns to the forefront and with the goal of improving agility, scalability, and resilience. Microservices and their relationship to integration are too complex to describe in this article.

Based on the following figure, when you look more deeply into the integration needs of digital teams, you see a need for something more than just a gateway to expose their APIs. Although these new applications and APIs can be crafted by using raw language run times, many of the composition challenges are well-known integration patterns, such as mapping and enrichment. Therefore, it makes sense to use simple tools and frameworks to accelerate the implementation of those APIs.

In addition to the external gateway for APIs, digital teams also have the following integration needs:

• API composition to enable the creation of more sophisticated APIs by aggregating and composing calls to existing APIs both within and beyond the enterprise.
• API and event discovery especially to discover and consume capabilities that are exposed by enterprise IT as seamlessly as possible, regardless of cloud boundaries.
• Dependent partner management to manage their relationships with external dependencies, handling subscriptions, connections, certificates, and credentials, and managing and monitoring usage as appropriate.
• Application-centric, lightweight messaging within their application environments, with Internet-centric capabilities, such as elastic scalability, low latency, high subscriber capacity, but also critically with the ability to be connected to the enterprise messaging environment.
• Integration PaaS for productivity of digital teams. Ideally these teams want to remove the task of building up a platform. They prefer to use an existing platform from the start and focus on the function of their applications and APIs.

Seeking consistency with flexibility

A hybrid integration architecture must enable frictionless integration across the enterprise, with the following qualities:

• Consistent and simple. Integration across application boundaries in consistent ways that use only the elements of integration that are needed in each context
• Hybrid aware. Integration capabilities wherever the applications are, whether on-premises, within the cloud-based infrastructure, or within a platform
• Agile at scale. Agility and productivity, with the flexibility to step up to the mantel of enterprise grade and internet scale

Conceptually, integration should feel similar for everyone, even though your specific needs might vary. To achieve these competing demands, you must look for ways to simplify the problem.

As an example, a common way to achieve consistency in an architecture is to look for repeating patterns and use them to simplify the architecture. In the architecture that is presented in this article, a repeating pattern is clearly visible across the hybrid integration space and consists of two core elements:

• Exposure, which is typified by a gateway that exposes APIs
• Implementation, which is typified by a run time in which lower-level interactions can be composed

These two elements are repeated in varying degrees across the integration landscape, although the depth and sophistication that are required from each of these two elements varies by scenario.

For example, in the lower layers that are shown in the following diagram, when you connect deeply to existing systems of record, you are likely to require the full toolbox of capabilities. Now, compare this arrangement to the upper layers of the diagram. Here, digital teams that expose new APIs based on the composition of existing internal APIs, and external partners, might require only the gateway and some of the more lightweight composition capabilities. Teams that create new applications might need little more than a gateway to expose their APIs and access the already exposed internal APIs.

This view of the architecture provides a look into a much broader set of opportunities for simplification. By targeting elements of consistency, such as the ones shown here, we encourage a common, but flexible pattern to perform integration. We simplify and rationalize the architecture without sacrificing capability. We aim to explore these areas of consistency with flexibility further in future articles.

Conclusion

This article explored the evolution of hybrid integration. It explained how hybrid integration has extended the ownership boundaries of an enterprise. It defined the core capabilities of hybrid integration and the role and requirements of the lines of business. Then, it demonstrated how the central IT team bridges the bi-modal divide, empowering the digital team. It differentiated between public APIs and internal APIs. Finally, this article demonstrated the importance of meeting the needs of the digital team and ensuring consistency with flexibility in the hybrid environment.

In this article, we deliberately avoid making references to IBM products, but as you can imagine, we have a strong point of view on them. Through the remainder of 2016, watch for posts on the IBM Integration Developer Center blog that map the reference architecture to IBM offerings.

Acknowledgements

Thank you to the following people for their input and review of the material in this article: Andy Garratt, Peter Broadhurst, Carsten Bornert, Guy Hochstetler, and Carlo Marcoli.