OSGi demystified, Part 1: Bundles of fun

The OSGi demystified series of articles addresses common OSGi issues in CICS, offering insight into OSGi, discussing best practices, and providing setup and configuration advice. Part 1 focuses on OSGi bundles and OSGi services. OSGi application and architectural guidance is also provided, alongside a round-up of useful materials.

OSGi bundles

As with all paradigm shifts, it can take a certain amount of education and persistence to gain advantage from a new technology. So if you are confounded by Java, bemused by bundles, or up-in-arms over OSGi, then read on. Let’s face it — the alternative could be that you become irrelevant, left behind, or even obsolete. We don’t want that.

If you are confident and raring to go, then by all means install CICS Explorer, add the Java/Liberty component, write some Java code, create an OSGi bundle, and deploy your application to CICS. For the rest of us, consulting the product documentation is a good start, see CICS Java support — just in case there is something there of use. You’ll also be needing CICS Java troubleshooting, the troubleshooting guide for OSGi JVM servers.

OSGi reminder

Open Services Gateway initiative, or OSGi, is a Java framework for developing and deploying modular software programs and libraries. The modular components of OSGi are known as OSGi bundles. Each OSGi bundle is a tightly coupled, dynamically loadable collection of classes, jars, and configuration files. Each OSGi bundle explicitly declares the packages they provide and the packages they require.

By enforcing explicit declarations, the OSGi framework can provide a powerful component model. Beyond the obvious benefits of componentisation and re-use, the framework uses install-time dependency resolution to help avoid run-time failures. OSGi also offers a powerful versioning mechanism where multiple versions of the same class can exist within the same JVM. Improved class loading performance is realised as a result of the clearly defined dependencies, and a dynamic service architecture allows applications to be updated without a server restart.

In CICS, the OSGi JVM server is a Java runtime that incorporates an OSGi framework. That same technology is used for the Liberty JVM server. The key difference between the two is that the Liberty JVM server embeds an instance of a Liberty server. Typically an OSGi JVM server is appropriate when Java SE APIs are required, and a Liberty JVM server is appropriate where web technology and/or Java EE APIs are required.

With those formalities out of the way, let’s move on to some clarifications.

OSGi bundles are not CICS bundles

You’ll be surprised how often these two types of bundles can be confused. CICS bundles are a package and deployment mechanism for CICS resources, while OSGi bundles are components of the Java/OSGi system.

Now that distinction is clear in your mind, let’s add some confusion. CICS bundles contain one or more resources called bundle parts. OSGi does not have any notion of parts, so throughout this text “bundle part” will always refer to a CICS bundle part. A number of bundle parts are relevant to CICS Java and Liberty, and we’ll discuss the Liberty-related bundle parts that correspond to WAR, EAR, and EBA in upcoming installments. For the moment, though, let’s concentrate on the pure OSGi solution, and that means the OSGi bundle part in an OSGi JVM server. It should be no surprise to learn that the OSGi bundle part allows CICS to deploy an OSGi bundle into an OSGi JVM server when the CICS bundle is enabled. OSGi environments outside of CICS have no concept of CICS bundles, so OSGi bundles in the wild are generally deployed directly to the OSGi framework. The CICS approach offers additional benefits, though, like the ability to control the lifecycle of both OSGi bundles and CICS resources within the same packaging mechanism (see Figure 1).

Figure 1. CICS bundle containing OSGi bundles deployed into an OSGi JVM server

CICS bundle containing OSGi bundles deployed into an OSGi JVM server

Bundles in Liberty

This section covers:

  • Enterprise bundle archives (EBAs) and their use in Liberty with OSGi bundles
  • The different ways OSGi JVM server and Liberty embrace library bundles
  • A quick explanation of Java- and Liberty-related CICS bundle parts

OSGi bundles in Liberty: EBAs

Given that a CICS bundle is essentially a container for one or more OSGi bundle parts, with each bundle part referencing an OSGi bundle, let’s double the confusion. Liberty also has a containment mechanism for OSGi bundles, called an Enterprise bundle archive, or EBA. Importantly, though, EBAs aren’t just a way to deploy multiple OSGi bundles, but also a way to deploy sets of OSGi bundles as a single isolated sub-system (think mini OSGi container per application). In practice, that means OSGi bundles in different EBAs are not visible to each other, by intention. Using sub-systems gives Liberty the ability to isolate applications from each other for both security and integrity purposes — but that’s a topic for another day. In the meantime, suffice it to say that CICS steps up to the mark and provides an EBA bundle part for deploying EBAs to CICS Liberty. Ultimately, your OSGi bundle sits inside an EBA, and your EBA sits inside a CICS bundle. It is usually these containment mechanisms, in tandem with unqualified uses of the term “bundle,” that cause outbreaks of bundle sickness. Now you have a cure.

Figure 2. CICS bundle containing an EBA deploying into a Liberty JVM server

CICS bundle containing an EBA deploying into a Liberty JVM server

OSGi library bundles: Not so common

Sooner or later, you’ll discover that applications tend to perform subsets of similar operations. Perhaps you’ll even develop common libraries and place them in distinct OSGi bundles. Doing so demonstrates that you’ve embraced the reusable component power of OSGi — well done! Your next challenge is to deploy them to your JVM server, making them accessible from all applications and ensuring that each component, or library, is installed and ready. It’s a great plan, with only one minor catch: Despite the fact that both OSGi JVM server and Liberty embrace library bundles, they do it in very different ways.

Recall that Liberty uses an OSGi application, or EBA, to create isolated sub-systems of OSGi bundles. An EBA can contain a set of JAR files (the physical manifestation of OSGi bundles), or it can contain a “shopping list” of OSGi bundle references. That list is essentially an index into something called a bundle-repository. You can configure a bundle-repository in Liberty’s server.xml. The bundle-repository is a directory on the file system that contains one or more OSGi bundles. When the shopping list is resolved, OSGi bundles are taken from the bundle-repository, installed into a common OSGi sub-space, and the EBA is given access to them. In this way, multiple applications can share common OSGi bundles while still retaining isolation for their main functions.

Note: If you deploy an OSGi application (EBA) within CICS bundles, the CICS Explorer tooling will upload all required OSGi bundles inside the EBA (the archive file). To use the shopping list approach, you must manually develop and deploy the OSGi application (EBA) and manually configure a bundle-repository.

In contrast, the OSGi JVM server shares a single OSGi space. Each OSGi bundle deployed to the OSGi JVM server is potentially available to all other OSGi bundles that have been deployed. Whether the OSGi bundles will actually place a dependency on each other or not is controlled through the Import-Package and Export-Package statements in the MANIFEST file of the OSGi bundle.

Ultimately, in an OSGi JVM server any OSGi bundle can provide common function, so the OSGi JVM server has a slightly different concept for a library which is known as a middleware OSGi bundle. A middleware OSGi bundle is one that shares the lifecycle of the container. These bundles are not deployed inside a CICS bundle; instead, they are installed and enabled when the OSGi JVM server is enabled. These bundles remain enabled for as long as the JVM server is enabled, and can be configured using the JVM profile option:

    OSGI_BUNDLES={comma separated list of path qualified OSGi bundles}

WAR, EAR, and EBAs

As promised, here’s a quick explanation of Java- and Liberty-related CICS bundle parts. An OSGi bundle part is used for deploying a single OSGi bundle to an OSGi JVM server. Liberty doesn’t allow direct deployment of OSGi bundles; instead, it requires a container into which one or more OSGi bundles can be placed, or referenced. We’ve already discussed the EBA bundle part in passing, and there’s not really much to add. Suffice to say that an OSGi application project created by your Eclipse-based WebSphere Developer Tooling will be exported (JAR’d up) into an EBA. A CICS bundle can contain an EBA bundle part, and when the CICS bundle is installed into CICS, the EBA is installed into the Liberty server instance running inside your Liberty JVM server.

In similar fashion, a (dynamic) web application typically created by the WebSphere Developer tooling in Eclipse (WDT) will be exported as a WAR file (Web ARchive). A CICS bundle can contain a WAR bundle part, and the web application is deployed to Liberty when the CICS bundle is installed. Unsurprisingly, that same pattern applies to the final artifact in our list, the Enterprise ARchive (EAR), which has a corresponding EAR bundle part.

Bundle misconceptions

This section looks at:

  • Differences when creating your JAR as an OSGi bundle
  • Misconceptions of OSGi bundles and application updates

An OSGi bundle is a JAR

There’s absolutely nothing to lose by creating your JAR as an OSGi bundle — even if ultimately you don’t deploy it to an OSGi environment. The key difference is that an OSGi bundle contains more metadata in the MANIFEST.MF in the JAR. This metadata offers at least three important pieces of information:

  • A unique ID (Bundle-SymbolicName and Bundle-Version)
  • A set of dependencies (Import-Package)
  • The external interface (Export-Package)

When not deployed to an OSGi environment, the metadata is simply ignored.

Figure 3. Example of an OSGi bundle MANIFEST

Example of an OSGi bundle MANIFEST

OSGi bundles allow versioning of code

Beware! This statement can give rise to misconceptions. The primary capability here is about allowing multiple versions of the same package/class to be available within the same runtime. Two different applications might have different version requirements on the same library. OSGi allows both versions to be deployed simultaneously, and both applications can be satisfied. However, take note: Multiple version support does not automatically provide seamless update of applications … and that leads us directly to the next point.

OSGi bundles do not provide dynamic update as well as you might expect

The misconception here is that you can deploy a new version of an OSGi bundle to the OSGi environment and everything magically starts using it. That could be true, but there are several “it depends” to navigate first.

For example, it depends on whether you rely on bundle-wiring (the act of resolving Import-Packages) or whether you are using OSGi services. If you are using bundle-wiring, it depends on where and when the previous version was referenced. If some other OSGi bundle (B) uses your OSGi bundle (A) and successfully resolves against it, OSGi will not resolve to the new version (A’) by default. If it did, there is a risk of breaking a working application. If yet another OSGi bundle (C) is installed, it might resolve to the new version of your OSGi bundle (A’), or it might resolve to the older version (A) — it is entirely dependent upon the way in which Import-Package dependencies are expressed. You can, of course, control the dependencies by judicious use of the version attribute on the Import-Package statement and by applying minimum versions and version ranges, but that’s hard work and very error prone.

To cut a complex set of interactions short, if you are using bundle-wiring (Import-Package) to dynamically update your applications, then you’ll have to be extremely careful with the order in which you install the new versions; you’ll need to be diligent about incrementing the OSGi bundle versions, and smart enough to figure out the right dependency ranges to ensure the whole chain is refreshed according to your wishes. My advice: Don’t do it — use OSGi services instead (see later). Alternatively, if you don’t mind a little downtime, you can always disable and re-enable the JVM server!

Version ranges and semantic versioning are good practice

Regardless of whether you use OSGi services or not, you should employ a suitable bundle/package versioning scheme that allows you to distinguish between bug fixes, updates (new function), and interface breakage (API incompatibilities). OSGi semantic versioning does just that, I encourage you to follow that model for your OSGi bundles. You can find a useful summary of this in the Semantic Versioning whitepaper from the OSGi Alliance.

However, knowing how to version your OSGi bundles is only half the story. Using version ranges in your Import statements on top of semantic versioning can make the difference between constant resolution problems with hidden breakages versus smooth transitions. For example:

a) Import-Packge: com.myorg.package1
b) Import-Packge: com.myorg.package1;version=1.0.0
c) Import-Packge: com.myorg.package1;version=[1.0.0, 1.0.1)
d) Import-Packge: com.myorg.package1;version=[1.0.0, 2.0.0)

Using (a), any version might match and this could lead to the classic pitfall of resolving against higher major versions. Doing this has the potential to break you at runtime because of interface changes.

Using (b), any version from 1.0.0 upwards might resolve and you will incur the same problems as (a). Most developers end up with a configuration like this because it’s usually the default tooling generated configuration.

Using (c) imposes a very restrictive match — the curved bracket implies that higher versions are excluded. This approach requires updates and redeployment of the application for every bug-fix in your chain of dependencies.

Using (d) is a happy medium. Bug fixes and API additions are tolerated without change. If there is an API breakage, the OSGi bundle fails to resolve and protects you from a potential runtime failure.

OSGi services

This section includes:

  • An explanation of OSGi services and the OSGi services registry
  • Declarative services and Service Component Runtime (SCR)
  • How to dynamically update with OSGi services and declarative services

OSGi services

With the revelation that dynamic updates done through bundle-wiring (Import-Package with version) are quite bothersome, it is time to talk about OSGi services. Shift your thinking from imagining that an OSGi bundle’s dependencies are on a particular package/version of code — and imagine instead that it depends on one or more services. Just as with package-dependencies, those services are provided by other OSGi bundles (though nothing stops you from providing them in the same OSGi bundle). The big difference is that services are dynamically registered with a central OSGi service repository rather than being declared statically as Import-Package statements in the MANIFEST.MF. The OSGi service registry is essentially a dynamic cache of all services, and provides a much more flexible way to facilitate component reuse.

An OSGi bundle can provide or consume one or more OSGi services. To be a provider, you should register your implementation of a service with the OSGi service repository — typically under a standard package/interface naming convention. To be a consumer, look up the service from the service repository and retrieve a service reference.

For any particular interface, there can be zero or more services registered. Unlike Import-Package and bundle-wiring, those services can come and go at any time. In essence, you can replace a service implementation with something better, stronger, faster, or fixed, as and when they become available. Of course it isn’t entirely zero-effort; you have to react to the loss of services and the appearance of newer implementations, and you’ll need a policy in place that determines whether you bind to a new service implementation, stick with the old one, or tolerate loss of a service for a while. OSGi offers various ways to handle that churn, from ServiceListeners to ServiceTrackers through declarative services (DS) and ultimately to something called Blueprint. For the purposes of brevity, we’re not going to discuss all of these approaches. Unless you really need fine-grained programmatic control and are a sucker for boilerplate code, you should go straight to declarative services (DS) instead. It is significantly cleaner, simpler, and more effective.

OSGi declarative services

Declarative services (DS) within OSGi are also known as the Service Component Runtime (SCR). Strictly speaking, the XML is the declarative services part while the runtime is the SCR — but the two are generally used interchangeably. The vast majority of interaction policies and component requirements can be expressed in DS’s XML syntax, giving you plenty of flexibility with minimal coding.

So armed with your new approach to using OSGi services and declarative services, how do you address dynamic updates? To start with, the bundle-wires, or package dependencies, are more stable because your dependencies are on an interface, not on a constantly changing version of an implementation. Under that interface, there can be one or more implementing services and you can have a DS policy that determines which service you use. That policy could be as simple as When a newer implementation of a service is available, bind to it, or perhaps Only bind to services from a particular Vendor.

Service implementations can be registered with an additional set of properties, and it’s those properties that can help to implement the example policies above. When you need a service, you can filter the matches based on a specific property or combination of properties, LDAP filter syntax is respected. You can also give services a priority and bind to the implementation with the highest priority.

The second compelling aspect of declarative services is that they are quite forgiving. When you remove an old implementation of a service and install a newer one, DS (or more precisely the SCR) has built-in buffering. It is that buffering that ensures requests for service don’t fail during the update window.

Further reading:

JRE class visibility, bootdelegation, and system.packages.extra

This section covers:

  • Loading of core JRE packages/classes
  • The OSGi framework and system bundles
  • Exceptions to the system bundles process that exposes known extension packages to the system automatically

In OSGi, loading of core JRE packages/classes (java.\*) is always delegated to the bootstrap class loader. The bootstrap class loader, unsurprisingly, is the class loader that loads/bootstraps the OSGi environment — don’t confuse it with Java’s boot class loader. In practice, there is only ever one JRE in the system, and so explicit dependency statements are not required to reach the core JRE classes. For that reason, it is never necessary to add a java.\* dependency to a bundle manifest. However, application bundles that require extended parts of the JRE must code an Import-Package statement; for example, vendor-specific extensions javax.\* com.sun.\* and com.ibm.\* require an import. This is because they are not delegated to the bootstrap class loader, and are instead treated as part of the OSGi system.

The OSGi framework provides a system bundle that exposes known extension packages to the system automatically. An application bundle registers its dependency on those extension packages by including an Import statement, just as it does for all other packages provided by normal OSGi bundles. The advantage of this approach is that extensions can be replaced with newer implementations by installing an OSGi bundle that contains the new code.

There is, however, one exception to this process. If a particular package is coded on the OSGi boot delegation list — by means of the org.osgi.framework.bootdelegation property — the package is delegated straight to the boot class loader. This action completely bypasses the OSGi class loading mechanism. Although it is convenient, because no Import statement is required to access boot-delegated packages, it restricts the flexibility of OSGi and is not considered a best practice for anything but core, single-version packages.

Occasionally, you may encounter vendor-specific extensions that aren’t automatically added to the system bundle by the OSGi implementation. Don’t be tempted to add them to the bootdelegation list. For these cases, and assuming the package is genuinely available from the vendors JRE, the property -Dorg.osgi.framework.system.packages.extra can be used to add the packages to the system bundle, and allow application imports to resolve. The following JVM profile excerpts show how one might use these properties:

  • Override CICS bootdelegation defaults (Warning: This replaces the existing list and could break CICS integration):

      `-Dorg.osgi.framework.bootdelegation=com.ibm.xtq.*, com.ibm.xylem.*`
    
  • Extend the CICS bootdelegation defaults (a more acceptable way to add packages to boot delegation):

      `-Dorg.osgi.framework.bootdelegation.extra=com.ibm.something.new.*`
    
  • Extend the system packages list to expose extra packages through the OSGi system bundle (best approach):

      `-Dorg.osgi.framework.system.packages.extra=com.ibm.xylem.types, com.ibm.xylem, com.ibm.xtq`
    

Summary

This article has explored a whole range of OSGi entertainments, from bundle-wiring to OSGi services, and class loaders to bootdelegation. There are many interesting gotchas in Java and OSGi! At CICS Java HQ, we use OSGi internally and hope that our experience can help you realise the full potential of your CICS Java applications.

There’s a whole lot more OSGi material to cover — be sure to check out the other articles in the OSGi demystified series!