Application Discovery - Group home

DevOps Integration Story between IBM Code Coverage and IBM Application Delivery Intelligence

  



About a decade after its inception, the concept 'DevOps' has been one of the hottest topics in the technology industry. Many project teams are releasing products at faster cycles with adoption of Agile methodologies. Timely and through testing of the pushed code has become a key to providing quality products to customers. IBM Explorer for z/OS Aqua(Aqua) DevOps team has been working on an exciting initiative which would help teams gather better insight into their product codes using IBM Code Coverage(CC) and IBM Application Delivery Intelligence (ADI). This blog post will outline the story of Z DevOps team with CC/ADI Integration, how it is implemented, and how it has changed the DevOps game in the teams that have adopted it.

The key goal of CC/ADI integration in Aqua is to use the power of analytics to improve on code quality of products by providing quantifiable and measurable data on code health. There are many ways to ensure quality of code, but many times users face the difficulty of finding a way to quantify this. Users would write unit tests in order to test their code, but often neglect the fact that their test cases might not be covering all the lines of code they have written. Code Coverage is a metric that can fill in the gap; it measures the percentage of code which is covered by the automated test suites. It is a straight-forward measure of counting number of statements in a body of code that is executed through a test run compared to the total number of executable lines. Code Coverage statistic gives the users deeper insight into the overall health of the code base by showing which of the executable lines are actually run through at least one test case.



Code Coverage is in line with Test Driven Development (TDD), a paradigm which encourages programmers to provide test cases for the functionalities prior to even writing a single line of code. In order to prevent a drop in code coverage percentage, a developer has to be conscious about writing test cases for any new lines of code added. Teams could set up a threshold on what percentage they would want to see their code coverage at and use that as a baseline to decide on whether or not their new release is covered by enough test cases. For example, if a threshold is set up at 20% code coverage, they would want to see a warning sign if the actual code coverage statistics comes short of that. Higher code coverage means more lines of code are tested within the product, providing increased stability and reliability on the product once released. 

Z DevOps team used a powerful IBM Code Coverage plugin developed by the IBM Z Common Components team. The Code Coverage plugin and its API provides the versatility to plug into any DevOps environment using different technology stacks. IBM ADI, a powerful cognitive DevOps data analytics solution, is used for data processing and visualization. It is an ideal candidate for processing and aggregation of the robust code coverage data collected by IBM Code Coverage Plugin.



IBM ADI(Application Delivery Intelligence) is a platform that uses the power of analytics to provide actionable insights and cognitively improve a team's DevOps activities. It provides important data on the code that can help developers, quality managers, project managers, DevOps engineers, and quality assurance teams to make critical decisions. ADI processes and aggregates raw data into presentable charts and graphs which will give the user a direct insight into current code health. It provides correlation data and historic data of any key DevOps metrics in a highly customizable UI to provide smarter insights into the user's daily DevOps activities. More information about IBM ADI can be found on the official website: https://www.ibm.com/ca-en/marketplace/application-delivery-intelligence





CC/ADI Integration process can be divided into different steps. Each of the jobs are scripted and configured on Jenkins, and are set up so that the completion of previous step would trigger the next step. 


    •  Generate Inputs




Generate inputs step gathers files and data that is required for the code coverage plugin to run: source code and compiled binaries of the code. In the process a probe script and a metadata file is also generated. The files are compressed into a zip file, and the resulting file is published to an Artifactory repository. Once the generate step is completed, the instrument step automatically kicks off. 


    •  Instrument (run) Code Coverage




On the instrument code coverage step, the script first verifies the testing framework used by the packages. (choices could be JUnit, RCPTT, etc.) The build job goes  through test cases, running them one by one, and the IBM Code Coverage plugin runs at the same time to record the result of the tests, while generating other useful data such as lines covered.


    •  Publish Code Coverage Output




Once the instrument job is completed, an xml report consisting of all the code coverage data that has just been collected is created, and is published on Artifactory as a zipped file. The result file is then uploaded to a specific location on RTC, our source control, which is then scanned nightly by the IBM ADI data provider. Once a new published result is uploaded, IBM ADI's data provider will pick up the result, analyze the data, and publish it as the newest report on its web GUI.




[caption id="attachment_14914" align="alignnone" width="640"] Architecture diagram of CC/ADI Integration[/caption]



ADI provides all the tools necessary for the users to implement the customized workbooks exactly in the way that the teams wanted them. Filters enable users to specify the packages or files that need to be included in the report, and select which data source to pull the data from. ADI enables administrators to set custom threshold for code coverage percentage so that the users would be able to set their own standards on what percentage code coverage they wanted to observe in their code. ADI provides historic analysis of data, enabling the users to compare the current statistics with any of the previously published code coverage results. Users can also use the 'benchmark' feature provided by ADI to select builds to compare. If a user has monthly release with daily builds kicking off on Jenkins, a user can select certain builds as 'benchmark builds', and ADI will generate a comparison between the selected builds. This feature comes in handy when a user wants to track a trend in code coverage in customized intervals. ADI also provides information such as 'minimum test cases required', a statistic which analyzes how many test cases are required to test certain method or function, to help users identify any redundant or obsolete test cases. 



[caption id="attachment_14376" align="alignnone" width="1489"] ADI Dashboard Screen shows an overview of a project's code coverage data as a snapshot. The bottom half of the screen shows historical trend[/caption]

 

[caption id="attachment_14378" align="alignnone" width="1666"] Detailed Source View shows code coverage individually as files, or aggregated as packages. It also shows information such as number of lines covered, new lines added, number of test cases needed, etc.[/caption]

 

One of the greatest values of this integration is focus in continuous improvement. Feedback and suggestions provided information on what kind of reports are more ideal, and helped to create custom reports catered to each team’s needs. Some teams wanted customized reports with focus on only certain packages within their code base. Some teams wanted the reports split up between different testing frameworks. (JUnit, RCPTT, etc.) Teams placed different meaning to the code coverage percentage provided by ADI, and started setting their own standards and benchmarks on what the threshold should be for 'good' code coverage. With more data available, teams gained access to construct reports that represent more accurate and relevant data for the all the users (developers, testers, quality managers, etc.). DevOps team works as intermediaries to deliver any feedback or enhancement request from users to the development teams, providing a unique angle on product quality. 




CC/ADI Integration provides users with tools to improve their daily DevOps activities by providing important analytical insight into their code coverage. IBM Z Systems uses the integration to generate code coverage data on multiple products, and use the data to help teams better improve quality of the products. Through this integration there was value added gathering valuable feedback to improve upon IBM products. CC/ADI Integration will be able to deliver even more value to the customers in the future as IBM ADI and IBM Code Coverage plugin continue to evolve with user feedback gained from this initiative.




This is a still ongoing initiative at IBM. If you want to know more about the initiative or our products, please send us an e-mail.

Aqua DevOps Team

James Kim - IBM Z Aqua DevOps Team - James.H.Kim@ibm.com

Kiril Streltsov - IBM Z Aqua DevOps Team - kiril@ca.ibm.com

Leopoldo Miranda - IBM Z Aqua Delivery Manager - polomm@mx1.ibm.com








ADI Team

Peter Haumer - IBM ADI Tech Lead - phaumer@us.ibm.com








Code Coverage Team

Alan Boxall - IBM Z Code Coverage Tech Lead/Architect -boxall@ca.ibm.com