Implement integration tests for cloud-native CLI tools using BATS and Jenkins
In this blog, you learn about an integration test implementation using BATS and Jenkins. This implementation framework provides the customizations required to create and run…
Recently, I ran into a problem where I needed a new integration test framework because of a growing list of features and supported environments of a command-line interface (CLI) tool I was using. The existing unit test framework did not work well for integration tests because it did not offer a viable solution for testing containers on Windows systems or testing deployments to Kubernetes.
To address this issue, I used an integration test implementation created by using various open source components, including a Bash Automated Testing System (BATS) and Jenkins. Both of these components provided the customizations required to create and run integration tests, making this implementation applicable to projects that use a CLI.
The cloud-native CLI tool I was testing was an open source tool used for iteratively developing cloud-native applications that could be deployed to Kubernetes environments. It supported multiple operating systems, including macOS, Windows 10, Ubuntu, and Red Hat. Updates to the CLI were provided frequently with new releases every two weeks, resulting in a need for automated integration testing to ensure stable releases.
I validated builds for the CLI through functional and unit tests run by a Travis continuous integration (CI) pipeline. To verify candidate release readiness, I ran integration tests before the release. The setup and running of these tests were performed manually (mainly due to limitations with Travis). Unfortunately, Travis does not support Docker on Windows and does not provide an easy way to verify app deployments to Kubernetes.
I found that executing the integration tests can be time-consuming, tedious, and a repetitive process, making automation ideal. The automated test implementation needs to verify that the CLI commands are executed successfully. It must also be able to launch and manage the execution of the CLI commands across multiple test systems running all the supported operating systems. The image below summarizes the implementation.
After exploring a few open source black-box test frameworks, I ultimately chose BATS. BATS provided an easy way to verify that CLI commands ran successfully with the ability to customize the framework. For example, shell commands in the BATS helper file were written to provide several features. First, the most recent version of the CLI can be built. Second, custom logging for test execution can be created. Lastly, several environment variables can be created to set up the Kubernetes deployment. The image below shows the console output after running BATS tests.
An implementation of the BATS framework solved some of the problems I discussed earlier. It provided automation to setup and test the latest CLI along with a way to verify deployments to Kubernetes. BATS is also supported across the same operating systems as the CLI. However, BATS does not provide a way to manage test execution across multiple test systems. To accomplish this, I used Jenkins.
Jenkins provided a user interface where jobs were created to run tests on systems. Since BATS is shell-based, it integrated well with Jenkins. Once the connections to the test systems were created and established in Jenkins, the jobs to run the tests were easily created with just a few shell commands. The following image shows the Jenkins user interface with the test systems running tests.
The integration test implementation flow was completed by doing the following:
First, a Jenkins job was executed on a test system using a simple shell command, meaning:
- The integration test repository was cloned to pull in the latest test cases.
- BATS was run to execute the tests.
- The test case execution status was monitored and reported.
Then, BATS ran the test setup commands in the helper file and the following occurs:
- The CLI and other required repositories were cloned to pull in the latest development code.
- The CLI binary was built.
- The logging of the test case output was configured.
- Environment variables were created to configure the Kubernetes deployment environment.
Lastly, BATS executed the integration test cases.
The integration test implementation accomplished the main goal of providing a means to automate the integration tests for the CLI tool, but I quickly realized some challenges and limitations remained. One challenge was being able to use virtual machines (VMs) for all the test systems. The Linux operating systems, Red Hat, and Ubuntu could be configured on VMs but Windows and macOS could not. The CLI tool required Docker, which is a virtualization technology. Running Docker on a VM is a “nested virtualization” configuration that is not supported by Windows. The current workaround for this issue is to use a Windows laptop as the test system instead of a VM. macOS is not a very popular operating system to install on a VM, so I used a local macOS laptop.
Setting up the test systems was also challenging. Along with the operating system and Docker, each system required several additional programs to be installed and configured. Some of these include git, BATS, Go, BASH, SSH, and kubectl. It’s important to note that some of these programs are installed and configured differently depending on the operating system, thus adding to the complexity of the solution.
Another challenge I faced was achieving 100% automation with the implementation. Currently, some manual steps are needed to successfully run the integration tests. One of the manual steps was authenticating into the Kubernetes environment on each of the test systems. This was required as the sessions timed out and required new login credentials. Another manual step was checking the logs on the test systems. BATS can be configured to display errors and runtime logs as console output, but this can be difficult when running many test cases in a single session.
Lastly, BATS does not support looping within a test case, so all the tests using similar commands need to have their own test case. This is not a major inconvenience until a test needs to be changed. With a change, all similar tests need to be altered. If the tests can be looped, just the single loop needs to be updated. I’m curious to know if anyone has found a solution to this limitation.
With the help of this integration test framework, I hope it’s easier for you to create and run integration tests on your next project that uses a cloud-native CLI tool. Hopefully, now you are inspired to begin, or perhaps continue, to use DevOps practices like this one to greatly improve efficiency through integration and automation.