Use DevOps to drive quality assurance and testing

Quality assurance and testing are essential if you want to develop and deliver reliable systems that meet and exceed your customers’ expectations. Properly testing software is something many companies struggle with. In fact, many times test teams cannot understand the technical complexities implicit in any comprehensive effort to test software. Even agile teams, which usually embed testing resources in the SCRUM, can struggle to achieve comprehensive and effective testing. Advanced techniques, including API testing and service virtualization testing, have advanced the art of QA and testing, but they require substantial technical expertise. What organizations really need is a comprehensive approach to QA and testing, including the technical capabilities to use the latest methods. DevOps principles and practices can help drive quality assurance and testing by improving communication and collaboration between all of the essential stakeholders. This article explains how to use DevOps best practices to transform your company into a quality-centric and successful organization.

Establishing a culture for quality

Quality requires not only a shift in behavior, but a dramatic shift in the organizational culture as well. Many companies initially achieve success by filling a need in the marketplace and rapidly expanding – only to find themselves unable to grow and adapt as market forces exert downward pressure on the firm’s profitability. Successful companies learn to grow and adapt by responding to competitive pressures and the global ecosystem in which the firm must operate. The most important factor, and often the most challenging, is to continuously ensure quality while also growing and evolving the services they provide. If an organization hopes to evolve its culture, then DevOps principles and practices are fundamental to its success. Quality management guru, W. Edwards Deming, pointed out many years ago that quality must be built in from the beginning. Establishing a culture for quality also requires that you understand the ecosystem within which the system is being developed.

Understanding the quality ecosystem

Most large-scale systems are built within a complex ecosystem that includes market pressure from both competitors and outside forces, including federal regulatory requirements. Many companies scramble to meet these demands while trying to maintain profitable. Developers typically rush to deliver new features using the latest technology with all of its challenges and risks. These demands for new features require rapid application development that is best accomplished using agile methodologies. Unfortunately, many agile teams struggle to provide adequate QA and testing resources to meet these demands. Ideally, SCRUMs have dedicated testing resources embedded in the team, but many large organizations have established organizational structures that require QA and testing resources to be separate from the development resources. In many industries, this separation is mandated by federal regulatory requirements. The relationship between the developers and the quality assurance testing function is less than optimal at times.

Dysfunctional QA

Developers focus on delivering new features while quality assurance and testing is responsible for ensuring that the code is free of defects. This paradigm is similar to the relationship between development and operations, where operations is responsible for ensuring the systems are always available.

A dysfunctional relationship between development and test teams occurs when the developer completes the code, often later than expected, and the delay in code delivery throws the quality assurance and testing process off-schedule. An even bigger problem is that QA and testing teams often lack the technical knowledge required to test the code at anything more than a functional level and are limited to using the interfaces provided in the application. There is often an unhealthy dynamic where the organization holds the test team responsible for “catching” bugs, without providing adequate tools and training to accomplish this goal. Yet, when an incident occurs that impacts the customer, QA is often held accountable and asked to explain, “why this defect was not found.” Some developers do provide automated unit testing which is necessary, although often insufficient.

Unit testing

Developers often write automated unit test cases. Sometimes they might even use test driven development (TDD) practices where the test code is created before the code being tested is created. Unit testing is essential, although limited in terms of scope. Test coverage based only on unit testing is rarely sufficient to ensure that the application is free of defects. Another important approach is continuous integration.

Continuous integration

Continuous integration (CI) is an effective methodology where changes from each developer are integrated, built, packaged and deployed to a development test environment. Most agile teams value continuous integration and rely upon it to identify code compile and integration issues immediately after the code changes are checked into the version control system. When problems are found immediately after checking in a specific changeset from one developer it is easier to identify and debug than when you have to sort through many changes checked in from a group of developers. Since the CI server is typically configured to build, package and deploy after changes from a developer are committed, the debug process is simpler and easier to accomplish quickly. This is analagous to the rationale behind continuous delivery.

Continuous delivery

Continuous delivery (CD) involves automated procedures which build, package and deploy changes as they become available. While deployed, these features are only accessible to an end-user at a specific time. Continuous delivery requires mature automated build, package and deployment procedures. This is often referred to as a deployment pipeline. Continuous delivery helps to ensure that the deployment process is a routine event, thus enabling organizations to deliver features more frequently – even one or more times each day. CD is particularly effective at improving quality because it usually results in smaller deployments. Risk is reduced and, if there are problems, they’re usually easier to address. Organizations that have mature continuous integration and delivery processes are more agile because they can rapidly deliver new features to customers and efficiently address defects when they occur.

Learn about continuous integration and delivery:

  • IBM Tivoli™ Provisioning Manager enables a dynamic infrastructure. It automates the management of physical servers, virtual servers, software, storage, and networks.
  • IBM Tivoli Provisioning Manager for OS Deployment automates remote deployment of operating systems.
  • IBM UrbanCode Deploy™ offers an automation deployment framework that reduces deployment errors and improves efficiency, correctness, and traceability.

Code reviews

Code reviews are one of the most effective methodologies for finding and removing defects in code early in the software development process. Similar approaches exist for reviewing documents, test cases, and even user stories.

Learn about code review

Continuous quality assurance and testing

Quality assurance (QA) and testing are full-lifecycle endeavors that are addressed throughout the entire software and systems lifecycle. Continuous integration and continuous deployment help by providing updated codebases for testing and, more importantly, updated test environments. Implementing effective quality assurance and testing processes requires a comprehensive framework that starts with a clear understanding of what needs to be tested.

Learn about testing:

  • IBM® Rational® Quality Manager, a web-based centralized test management environment, provides a collaborative solution for tracking and metrics reporting.
  • IBM® Rational Integration Tester is part of IBM® Rational® Test Workbench, which combines the capabilities of functional, performance, regression, load, and integration testing, and automated testing of highly complex applications.

What we are testing

Testers need to understand the business itself and the requirements for the system being verified. From a DevOps perspective, testers should partner with business experts, including the product owner, to understand how the system being tested needs to function in order to support the business. The first step is to create functional tests.

Functional testing

Industry-strength functional testing tools help to ensure that functional tests are repeatable and also help ensure that there is maximum test coverage. Unfortunately, functional testing is often limited by the features that can be accessed via the user interface. Writing good functional test cases is often challenging. Fortunately, feedback loops exist throughout the lifecycle. These loops help to identify potential test cases that could prevent defects from impacting future production releases. Feedback loops are a key resource within DevOps.

Feedback loops

Feedback loops can range from the Service Desk report of a user-discovered bug to formal incident and problem management processes. When issues occur, it is always essential to share this information with the quality assurance and testing team because additional test cases might be required to ensure that defects are not delivered in a future release. Retrospectives also provide an essential opportunity to provide feedback to improve the testing process, this includes addressing any problems which occurred. Too often these issues cannot be tested using functional testing procedures and a more technical approach to testing is important.

API testing

Many applications today provide a Restful API which can be used for detailed and comprehensive automated testing. Testing using an API requires significant programming skills which many QA and testing professionals do not have. Even technically skilled quality engineers might not have enough in-depth knowledge of the application to effectively develop automated tests using a Restful API. The DevOps practice of pairing skilled and disciplined testers with developers who know the code is an effective approach to developing robust automated tests with a Restful API. However, message protocols can still be difficult to discover.

Discovering message protocols

Many systems are so complex that even skilled software developers find it challenging to identify all of the potential message protocols that can be found when systems are used in production. Tools and procedures exist today to monitor messaging protocols. The resulting information is used to develop robust test cases that can exercise features or behaviors that might not be easily accessible from the existing user interface. This works when a tester exercises the system while a technical resource utilizes a proxy to monitor and capture messages which, in practice, might be displayed in XML, JSON or some other representation. The messages can be copied and modified to provide additional test cases used to provide more test coverage. Even with robust test cases developed at the API level, many organizations are constrained by the availability of testing environments. This is where service virtualization testing can be of significant value.

Service virtualization testing

Testing environments are often expensive and may have limited availability. Service virtualization testing addresses this requirement effectively. Service virtualization testing typically involves recording a test case using a proxy which monitors messages at the API level. The messages can be played back, effectively mocking the test resource. For example, you might have a limited time to test your new system features against the IBM mainframe which is only available for a short period of time once a week. Recoding the messages exchanged between the tested application and the virtualized test resource allows for a virtualized test environment to be used when the mainframe is not available. Detailed API and service virtualization testing requires significant expertise. The DevOps approach of pairing disciplined QA and testing resources with their development counterparts can help to empower the organization to take a strong focus on learning and knowledge management.

Empower testers with knowledge

Testers who are paired with developers have an excellent opportunity to gain significant technical knowledge. The developers themselves learn about the art and discipline of testing. The DevOps focus on improving communication and collaboration can help the organization to develop a culture where knowledge is routinely shared and valued as a key strategic resource.


DevOps provides a powerful set of principles and practices that help improve communication and collaboration between the organizational silos which too often exist between the QA and testing organization and their development counterparts. While many agile teams embed testers in the SCRUM, others may require an organizational separation between these teams. DevOps best practices can help these technology professionals share their knowledge, experience and specialized skills to deliver quality systems that delight their customers and ensure success and profitability.