In a previous blog post, I discussed the importance of embedding security controls into development workflows through DevSecOps paradigms. Although these shift-left security controls in DevSecOps are absolutely essential, they are not sufficient to harden and maintain the security and compliance state of a cloud-native application. There are various security characteristics that manifest only during application runtime or when you shift right. For example, illegitimate file or process mutations inside a pod, denial-of-service, or network access anomaly all manifest when an application is live.
In other words, you can think of compliance as the desired state for your cloud-native application. The desired state is typically the compliance policy for every organization and could be different for every user. By having well-defined DevSecOps controls (such as code scanning for vulnerabilities or CIS controls) in CICD pipelines, and enforcement gates (such as a Kubernetes admission controller) at the cluster, you can ensure to attain and attest to the desired compliance state for your applications on day 1 as they are deployed on a cloud. Once your application is live, there are a number of factors that affect the compliance state, as shown in the following chart.
This diagram shows the typical lifecycle of an application from source code on the left to running workload on the right, processed through a CICD pipeline. Further, it depicts 4 primary factors that affect the compliance state of an application, which are:
- Malicious attack on a running workload, such as changing network policies on the cluster, “exec-ing” into the pod, and starting a malicious process.
- Change in the desired state itself when the CISO changes a regulatory policy. For example, if the maximum number of admins for a database is changed from 3 to 2.
- New threat intelligence is discovered every day that affects the existing compliance state. For example, a new CVE is reported and it affects one of your existing software packages.
- New bug or change request is submitted by a developer.
All these factors attributes to the fact that DevSecOps controls in CICD are point-in-time certifications of compliance state and it’s necessary to have a continuous control loop that ensures the desired compliance state is duly managed and maintained.
Code 2 Container (C2C) is our vision at IBM Research to federate and bring security measures across CICD and cloud runtimes together to provide continuous, complete, and effective compliance management for cloud-native applications. In C2C, we established two primary objectives: (1) code provenance profiling and (2) incident scope and control. Let’s dive into these two goals in detail.
Code provenance profiling
Following a strict DevSecOps practice has a few foundational criteria:
- Every cloud resource has associated source artifacts in a code repository. For example, every application pod has its source code, build artifacts (such as Dockerfiles), and deployment artifacts (
*.yaml), every network policy has associated
policy.yaml, and every object store bucket has its Terraform manifests.
- All of these source artifacts are scanned and validated through security and compliance controls in CICD pipelines.
- Only the vetted artifacts are used to provision and configure cloud resources.
In C2C, we aim to provide complete visibility into such complete code provenance, so that all relevant artifacts for our application can be viewed through a single pane of glass. For example, the state of a code repository with a specific
commitid is linked to an instance of a continuous integration (CI) pipeline (such as a Tekton
PipelineRun) that implements all DevSecOps security controls, which in turn are linked to the evidence report about the compliance verdict produced by respective controls. Finally, the link is extended with an instance of a continuous delivery (CD) pipeline and to the identifier of the target cloud resource.
An example of such provenance tracking is shown in the following diagram. This diagram highlights key properties that need to be captured through the software delivery lifecycle to ensure unique provenance tracking.
An important notion that underpins an effective provenance profiling is that we identify every touchpoint in the complete CICD workflow that either performs a code transformation (for example, in a CI build stage, code is transformed into an image) or produce new artifacts (such as an evidence report for vulnerability or CIS security controls). Then we perform specialized instrumentation of these touchpoints to record their activities in our C2C store through beacons. These beacons essentially are small tasks that run in the context of the CICD pipeline, gather required details, and emit to the
c2c-store. From there on, at any point in time, we can select any component from the provenance link and identify all associated states and artifacts. For example, for workloads running in a cloud, we can precisely view the state of the code (
commitid), validation of DevSecOps scanning, and associated compliance state. Another corollary objective we satisfy with
c2c-store is for controlled change management, wherein every code or state change can be effectively profiled and managed.
The following diagram shows the control loop that needs to be implemented to ensure continuous compliance profiling.
Although DevOps bridged the gap between development and operations teams for delivering software (code built into an image and then instantiated as a container) on the cloud, it does not address the problem of lack of synergy between Dev and Ops for effective remediation of workloads running on the cloud. Once a cloud-native application is running in production, my team employs security monitors to continuously check for any security or compliance violations. As I previously mentioned, these monitors can be part of periodic DevSecOps pipelines (shift-left activities such as continuous code scanning for vulnerabilities or deployment misconfigurations) or run alongside workloads on production clusters (shift-right activities such as StackRox evaluation of network segmentation or deployment risk based on network monitoring).
Again, in C2C, we provide that single pane of glass, the common context for managing any security issue. Once an incident is identified by monitors, first we create a scope for that issue, which essentially is a virtual ensemble of all associated states and artifacts related to that issue. For example, when our periodic vulnerability scanning pipeline job discovers a new vulnerability, it tags the associated state of code repository, Docker image built from that code, all running workloads from that image, and cluster configuration for that workload. For example, if the workload is exposed through an ingress. Similarly, when monitors discover that any security alters at the runtime, an issue is opened against the code repository that hosts the associated code with all of the scoping details. The incident scope records are derived and persisted in
The incident scope thus becomes a driving function in scheduling and prioritizing remediation actions of these security issues. For example, when we discover vulnerability in the code, the additional context that this code is running in production in the pod which has open ingress, helps us increase the priority for fixing that vulnerability. It also helps us track the progression of remediation. For example, the vulnerability is fixed in the code, processed through CI, and successfully built into an image, but the image failed to deploy. Therefore, C2C incident scoping helps synergize Dev and Ops in effective remediation and helps the security officer with efficient and continuous compliance management.
In summary, Code 2 Container is a design, an approach to federate and unify various security functions during the software delivery lifecycle under common context. This facilitates better visibility and an automated closed-loop security and compliance management for cloud-native applications.