Getting to know the Hyperledger Fabric continuous integration (CI) process
Explore the tools and workflows that the Hyperledger Fabric community uses for continuous integration and delivery
As with many collaborative open source projects, being agile while delivering quality code requires a well-defined process, allowing the contributions from developers to be swiftly integrated. With numerous code contributors from around the world all making contributions to projects that are developed in parallel, the Hyperledger Fabric community leverages various tools and workflows for continuous integration (CI) and delivery. This article introduces you to these the tools and discuss the workflows that the Hyperledger Fabric community uses while working with the Hyperledger Fabric CI infrastructure.
This how-to article will be most valuable to current contributors to Hyperledger Fabric, as well as new contributors looking to gain a better understanding of the processes used to build, test, and release Hyperledger Fabric. A familiarity with agile software development, lean software development practices, and continuous integration/delivery is key to understanding the concepts discussed here. In addition, you should be knowledgeable about Hyperledger Fabric and the projects associated with it. If this is your first time learning about Hyperledger Fabric, I encourage you to read through and try out the examples provided in the Getting Started with Hyperledger Fabric docs.
NOTE: As with all software projects developed in an agile fashion, the tools and processes discussed in this article are subject to change. At the time of this writing, the content discussed here is accurate through release 2.0 of Hyperledger Fabric.
The material covered here should take roughly 30-45 minutes to complete.
Discovering the Hyperledger Fabric projects
As the Hyperledger Fabric documentation states, “Hyperledger Fabric is a platform for distributed ledger solutions, underpinned by a modular architecture delivering high degrees of confidentiality, resiliency, flexibility and scalability. It is designed to support pluggable implementations of different components, and accommodate the complexity and intricacies that exist across the economic ecosystem.”
With that, let’s begin by reviewing all of the projects in Hyperledger Fabric:
|Project||GitHub (Read-Only Mirror)|
Note: There are a few miscellaneous Hyperledger Fabric repositories not mentioned in the above table. For the full list of projects, see https://github.com/hyperledger.
Numerous developers contribute to each of these projects on a daily basis. In order to ensure that each contribution maintains the integrity of and provides value to each Hyperledger Fabric project, the community uses Jenkins, an open source automation server to create builds, run tests, and orchestrate releases for each Hyperledger Fabric project. The Jenkins CI infrastructure is provided to the Hyperledger Fabric community by the Linux Foundation, and in cases where maintenance and administrative tasks are required to support it, the Fabric CI team is responsible for resolving these issues. If you ever have questions about the community’s implementation of Jenkins, the best place to get help is in the #fabric-ci channel on rocket.chat for continuous integration discussions, and the #infra-support channel for general infrastructure discussions.
Note: Rocket.Chat is an open source tool that’s used for team collaboration, similar to Slack and HipChat.
Identifying key deliverables
Hyperledger Fabric is an open source, distributed ledger technology (DLT) platform that comprises of several key components:
- Hyperledger Fabric — The core distributed ledger project
- Hyperledger Fabric CA — An optional certificate authority service that generates certificates to configure and manage identity in your Hyperledger Fabric network
- Official Hyperledger Fabric SDKs:
- Hyperledger Fabric Node SDK — Allows Node.js developers to use APIs to interact with a Hyperledger Fabric blockchain network
- Hyperledger Fabric Java SDK — Allows Java developers to use APIs to interact with a Hyperledger Fabric blockchain network
Each of these projects has key deliverables that are included with each release. At the center is the Hyperledger Fabric project, which is itself comprised of multiple components. The Hyperledger Fabric community publishes Docker images for each component on its public-facing Docker registry. They include:
Core Docker images
In addition to the Docker images that are published, the Hyperledger Fabric community maintains:
- A set of platform-specific binaries for each of the core components required to build a Hyperledger Fabric blockchain network.
- With the Fabric SDK Node project, the Hyperledger Fabric community also publishes two NPM modules:
fabric-client— Applications can use this NPM package to install and instantiate chaincodes, submit transactions, and make queries against a Hyperledger Fabric-based blockchain network.
fabric-ca-client— An optional component that allows applications to enroll peers and application users to establish trusted identities.
The build process
In this section, we’ll discuss the build process for the core project, Hyperledger Fabric. Generally speaking, all of the Hyperledger Fabric projects follow a similar build/test/release process. For the sake of brevity, I’ll focus on the main Fabric project in detail.
The developers in the Hyperledger Fabric community submit code to the project on a continuous basis. Code submissions are prepared and submitted for inclusion consideration via Gerrit, an open source code review tool that allows community developers to discuss project changes and manage git repositories and workflows. When a Gerrit patch set is submitted to the Fabric repository, the Jenkins server triggers the CI build process to test and validate the Gerrit patch set. Next, let’s take a quick look at the Jenkins dashboard.
The Jenkins dashboard lists all of the Jenkins jobs required to build, test, verify, and release each of the Hyperledger Fabric projects. It is organized and displayed in separate views. You can visit the Fabric CI Jenkins dashboard at https://jenkins.hyperledger.org/.
Note: You do have the ability to login and view the Jenkins job configuration details by using your Linux Foundation ID. For more information on creating a Linux Foundation ID, visit https://identity.linuxfoundation.org/.
There are more than 100 different CI jobs, each broken down into various categories. For example, take a look at the Fabric tab on the Jenkins dashboard, which displays each of the Jenkins jobs related to the Hyperledger Fabric project:
Figure 1. Jenkins dashboard – fabric tab
When a Gerrit patch set is submitted to Hyperledger Fabric, the first job that executes is the fabric-verify-build-checks-x86_64 job. This Jenkins job is responsible for building and publishing the Docker images and the Fabric binaries that include the newly submitted code. These images and binaries are later pulled (or downloaded) in subsequent Jenkins jobs, when a number of event conditions are met in the fabric-verify-build-checks-x86_64 CI job. Figure 2 illustrates the current verify job process:
Figure 2. Fabric verify job workflow
(Source: Hyperledger documentation)
This illustration is from the Fabric CI documentation page and captures the Fabric verify job workflow as of the 1.3 release of Hyperledger Fabric. When a developer submits a patch to Gerrit for code review, the patch is made available for review by the other developers inside the community. Although automation provides the community with a consistent, reliable way to build, test, and release code, there is still a lot of value in having your code evaluated with “human” observation. Oftentimes, this allows for commentary about the change and presents the community with the opportunity to discuss potential improvements, mistakes, or alternative ways to implement the feature or fix.
In parallel with the code review process, Jenkins is triggered to run as a series of tests. (I’ll cover the tests in detail below.) Once these tests pass, the newly built Fabric images and binaries are published to the Hyperledger Nexus 3 and the Hyperledger Nexus 2 repositories, respectively. Nexus is a repository management tool that’s used for storing the build artifacts from each of the Jenkins build jobs, and allows you to publish new build artifacts on a daily basis.
Testing Hyperledger Fabric
It is important for Hyperledger Fabric contributors to understand the testing process. Developers can get confused about why their contributions fail the QA workflow. This section introduces the test processes for Hyperledger Fabric and describes the kinds of tests that are required to ensure the highest level of code quality across the community.
Figure 3. QA testing pyramid
(Source: Hyperledger documentation)
The graphic shown here is from the Testing Info documentation page included inside the Hyperledger Fabric repository. It illustrates at a high level what the QA team strives for when testing Hyperledger Fabric. At the base of QA testing, you start with more isolated, focused testing of individual components. As these tests pass, the workflow moves on to broader, more integrated testing, ensuring that code changes not only work for their function, but cleanly integrate with all of the other components and features of the system.
In Figure 2 above, you see that the Jenkins pipeline triggers a number of jobs to fully test the Fabric codebase. Let’s take a high-level look at the various job/test types:
- Verify jobs are triggered at patch set creation. Verify jobs act as an upstream Jenkins job that runs the initial build process, and trigger a series of downstream jobs that run the Hyperledger Fabric test suite.
- Merge jobs are triggered when a patch set is merged into the repository.
- Release jobs are on a
ref-updated-eventevent, which typically happens when a new Git tag is created to indicate a new release in the repository. You’ll see more about this process in the release process section below.
The tests that run against the Hyperledger Fabric codebase are determined by the type of Jenkins job that is triggered. Generally speaking, verify jobs tend to run more of the linting and unit tests, whereas merge and release jobs typically run more of the integration and full suite tests.
Supported test types
As of Hyperledger Fabric release v1.3, the verify job triggers a number of downstream Jenkins jobs that perform a series of tests against the newly built codebase. The tests types are as follows:
- Unit tests
- Integration tests
- Interoperability tests
- System tests
In Hyperledger Fabric, unit tests validate isolated functionalities of the project. In addition to the Fabric CI pipeline, developers also have the ability to run unit tests, as described in the Running the unit tests documentation.
Referencing the integration test documentation, these tests cover testing between components — for example, tests between peers and an orderer — and are used to show that a feature is working as expected across different interfaces. They ensure that components and systems work together as expected for a specific feature.
Interoperability tests are higher-level integration tests that span across repositories to test the compatibility between the Fabric, Fabric-CA, and Fabric-SDK projects.
System tests are longer running tests that perform the following validations:
- Performance and concurrency
- Upgrades and fallbacks
- Long running
- Chaos and recovery
- Functional (only for regression purposes)
As mentioned above, for more details on all of the tests performed, see https://github.com/hyperledger/fabric/blob/release-1.3/testingInfo.rst.
If you want to add new tests to the codebase, the How to Contribute documentation from the fabric-test repository describes how you (as a developer) can use behave tests to write custom tests for Hyperledger Fabric, including how to:
- Write custom feature files
- Write Python test code to execute the feature files
- Add docker-compose files for different network configurations
The Hyperledger Fabric community aims to perform releases on a quarterly cycle. Before a new release is published, a release engineer (RE) has to check a number of prerequisites for each of the Hyperledger Fabric repositories.
The release process is executed by running the release process for the following Hyperledger projects (listed in order):
On release day, the Fabric CI team checks that the latest nightly builds have completed successfully. Upon verifying this, the Fabric CI team identifies each of the commits and manually runs the release jobs that build and tag all of the build artifacts (whether they are Docker images or platform-specific binaries) as the Fabric release commits. Finally, the Hyperledger release managers publish the release images and binaries to the Hyperledger Dockerhub registry and the Nexus repository for public use.
Contributing to Fabric CI
With several projects being contributed to by community developers across the globe, it’s important to have a reliable yet adaptable Jenkins pipeline to provide continuous integration tools to the community. And just as any open source project would do, the Hyperledger Fabric CI team encourages developers to get involved. If you’re interested in contributing to these efforts, this section will help you get started.
First, take another look at the Jenkins environment provided for the Hyperledger Fabric community. The Fabric CI environment is actually comprised of two parallel deployments of Jenkins:
- Jenkins Production, located at https://jenkins.hyperledger.org/
- Jenkins Sandbox, located at https://jenkins.hyperledger.org/sandbox/
Note: You will notice that both of the Jenkins production sandbox instances are accessible from the same URL, though the sandbox instance is accessed at
Jenkins Job Builder
Keeping up with the tasks that each Jenkins job performs can become increasingly difficult as the number of jobs (and tasks each job performs) grows. This is why the Fabric CI team leverages Jenkins Job Builder, which takes job configurations described in YAML format and uses them to configure Jenkins. This way, the Jenkins job configurations (YAML files) can be stored (and version controlled) inside a repository.
With each of the job configurations written in YAML format, the Fabric CI can easily make modifications to Hyperledger Fabric CI jobs to meet the needs of the community. The ci-management repository is where the jobs configuration data is stored for all of the Jenkins jobs that the Hyperledger Fabric community employs. Just as the great Hyperledger community does, the Fabric CI team utilizes Jira, an issue and project tracking tool provided by the Linux Foundation to identify CI-related tasks to work on and bugs that need to be fixed.
Jenkins Sandbox environment
When Jenkins jobs are misconfigured the process is broken, which leads to failed jobs that can prevent Fabric images, binaries, and even the documentation from being published. This is why it is very important to maintain a high level of confidence when it comes to the CI pipeline process. To stage changes to the Fabric CI jobs, the Fabric CI team leverages a sandbox environment that allows the CI team to test Jenkins job configurations prior to having them merged into the ci-management repository, thereby applying these changes to the live production Jenkins environment.
As mentioned in the Fabric CI Sandbox setup documentation, there are several things to keep in mind prior to working in the Hyperledger Jenkins Sandbox environment:
- Jenkins jobs in the sandbox environment are automatically deleted every weekend.
- Fabric CI contributors can login and configure Jenkins jobs in the sandbox environment directly.
- Sandbox jobs cannot publish images to DockerHub.
- Sandbox jobs cannot vote on Gerrit patch sets.
- Jenkins nodes in the sandbox environment are configured using Openstack VMs that cannot be accessed directly.
Keeping these things in mind, using the sandbox environment to test your CI jobs helps ensure that the production CI environment is reliable. If you’re interested in contributing to this effort, here’s a summary of what you’ll need to do to use the Jenkins Sandbox environment:
- Clone the ci-management repository.
- Install the Jenkins Job Builder.
Duplicate the Jenkins Job Builder configuration file (jenkins.ini), then modify the settings in your local copy to push changes to the Jenkins Sandbox host. The file you will modify should look like this:
[job_builder] ignore_cache=True keep_descriptions=False include_path=.:scripts:~/git/ recursive=True [jenkins] user=johnquser password= <Refer below steps to get API token> url=https://jenkins.hyperledger.org/sandbox This is deprecated, use job_builder section instead ignore_cache=True
- Retrieve your Jenkins API token from the Jenkins Sandbox host. Place your API token as the value for the
password=variable in your local copy of the jenkins.ini file.
At this point, you’re ready to make contributions to Hyperledger Fabric CI. From the ci-management repository, you can make changes to add/remove/modify build steps defined across a number of Jenkins Job Builder configurations to support the needs of the community. Before you submit your modifications for consideration through the code review process, you want to ensure that your modifications are valid. You can test your Jenkins job configurations with the following command:
$ jenkins-jobs --conf jenkins.ini test jjb/ <job-name>
A successful test will display the Jenkins job configuration (in XML format) that will be imported by Jenkins. This is a good indication that your job modifications will work on the Jenkins host. The next step is to actually run your updated Jenkins job in the sandbox environment. You can now execute the following command to update/push your job modifications to the sandbox:
$ jenkins-jobs --conf jenkins.ini update jjb/ <job-name>
If your Jenkins Job Builder configuration file is configured properly, your newly updated Jenkins job will now exist on the Jenkins Sandbox host.
Test your new job on the Jenkins Sandbox host by selecting the job that you want to trigger, then click Build with Parameters, and finally click Build.
Observe the console output from the Jenkins job to validate that the job is executing the expected steps.
For a more detailed description of this process, I would encourage you to read through the Fabric CI Sandbox setup documentation located at https://github.com/hyperledger/ci-management/blob/master/Sandbox_Setup.md.
By now, you should have a solid understanding of how continuous integration and delivery methods are used to build, test, and release the Hyperledger Fabric projects. Without the tools and workflows implemented by the community in place, the projects adopted by Hyperledger Fabric would not be developed with the agility it requires to swiftly improve the code or react to customer needs. As a member of the Hyperledger Fabric CI team, I would encourage you to contribute to the community, participate in our weekly scrum meetings, ask questions, and provide us with valuable feedback as we continue to deliver improvements to our processes and tools.