DevOps brings development and operations together. It enables businesses to continuously develop and deploy services and offerings on the cloud, incorporating customer feedback and new requirements as they arise. Security must be incorporated into this approach from the first stages of development – ensuring the application runs on a safe platform, the code is free from vulnerabilities, and the operational risks are clearly understood.
This article examines the five facets of secure DevOps, shows you how to implement them in your cloud solutions, and describes how IBM implements those standards when developing cloud offerings.
Five facets of secure DevOps
When creating a secure cloud offering, you should integrate the following facets into your DevOps process:
- Secure engineering to ensure the products and services are developed and built with strong security and privacy controls and operate in compliance with accepted international, national, governmental, industry, and regional security standards.
- Secure deployment and operations to ensure that cloud platform, runtimes and applications are deployed securely, checked regularly for security configuration and hygiene, tested for security vulnerabilities, and updated with software patches and security fixes.
- Separation of duties to ensure users only have the access that is required to perform their jobs according to the principle of least privilege.
- Availability and business continuity management to ensure your infrastructure, runtime components, and management components are highly available.
- Security evaluation and learning to ensure that the security functions and properties in the delivered code and services are maintained as threats evolve and new vulnerabilities arise.
Cloud infrastructures, cloud platforms, and cloud applications need to be engineered to address the threats found within the operational environment.
The core practices for secure engineering are:
- Security requirements and design
- Threat modeling
- Secure coding and integration
- Security testing
- Vulnerability and incident management
- Operational controls validation and improvement
Security requirements and design
To build secure software, you must start with clearly defined security requirements. Secure software is realized when these requirements are applied in design. Security in design drives security in coding, integration, delivery, and operations.
Security requirements and design is a development-centric practice that sets the standard for security for the software, system, or service being built.
Secure software can include or integrate with functions that are part of the end user experience, including secure logins and authorization and entitlement verification. Secure software can also include or integrate with functions that are invisible to the end user, including protecting data in transit or at rest, detecting and reacting to threats and attacks, and more.
Security requirements should reflect both end user security functions and system security functions. These requirements should be carried through design of software and the design of security controls that support secure operations. Security requirements should also be expressed in ways that users of your applications can recognize. A general model for operational security controls follows the IBM Security Blueprint as represented in the figure below.
Figure 1. Using the IBM Security Framework and IBM Security Blueprint to realize business-driven security (source for image)
Aligning security controls with security standards
You need to align your application’s security standards with your organization’s security standards and with the software users’ requirements. Depending on your country, industry, and the audiences your software serves, any of several international, national, governmental, industry and regional security standards may apply. These standards specify security functions and implementation techniques in software or in the platform and infrastructure in which that software operates.
Selecting a security-specific standard is based on a combination of:
- Who deploys and operates the service
- The user community served
- Where the deployed service operates
- What business or industry process the service supports
- What types of data are created, stored, or transmitted
Software development teams need to consider the following scenarios:
- The software or cloud service has a single industry and single purpose
- The software or cloud service is considered as a cross industry or general purpose
Security standards for single-purpose software can be obvious, for example, the US government, EU government, banking, healthcare, regulated utility, or payment card industries. General purpose or cross industry offerings might need to align with multiple standards.
Visit the Security Governance and Compliance section to view a complete list of well-known and recognized security standards.
Common security controls
To avoid over-thinking the question of which security standard(s) apply, we recommend selecting a set of security services and related security controls that are compatible with a number of well-known and well-accepted security standards.
The Foundational Security Services in the IBM Security Blueprint, along with a merged set of ISO27002 and US NIST SP800-53R4 represent a recommended starting point.
Table 1. Security functions and controls
|Security function||ISO27002 controls||NIST SP800-53R4 controls|
|1. Software, system, and service assurance||ISO 27002-2013 14.1.1|
ISO 27002-2013 14.1.2
ISO 27002-2013 14.1.3
|PL-2 System Security Plan|
PL-8 Information Security Architecture
SA-3 System Development Life Cycle
|2. Identity, access, and entitlement management||ISO 27002-2013 9.2.1|
ISO 27002-2013 9.2.2
ISO 27002-2013 9.4.1
ISO 27002-2013 9.4.2
ISO 27002-2013 9.4.3
|AC-2 Account Management|
AC-6 Least Privilege
AC-7 Unsuccessful Logon Attempts
IA-2 Identification and Authentication (Organizational Users)
IA-4 Identifier Management
IA-5 Authenticator Management
IA-8 Identification and Authentication (Non-Organizational Users)
|3. Data and information protection management||ISO 27002-2013 8.2.1|
ISO 27002-2013 8.2.2
ISO 27002-2013 10.1.1
ISO 27002-2013 10.1.2
ISO 27002-2013 12.4.2
ISO 27002-2013 13.2.3
|RA-2 Security Categorization|
MP-3 Media Marking
SC-8 Transmission Confidentiality and Integrity
SC-12 Cryptographic Key Establishment and Management
SC-13 Cryptographic Protection
SC-17 Public Key Infrastructure Certificates
|4. Threat and vulnerability management||ISO 27002-2013 13.1.1|
ISO 27002-2013 13.1.3
ISO 27002-2013 18.2.3
|SC-7 Boundary Protection|
CA-2 Security Assessments
|5. Command and control management||ISO 27002-2013 12.1.2|
ISO 27002-2013 12.4.1
|AC-3 Access Enforcement|
AC-4 Information Flow Enforcement
AU-9 Protection of Audit Information
CA-6 Security Authorization
CA-7 Continuous Monitoring
SI-4 Information System Monitoring
|6. Security policy management||ISO 27002-2013 5.1.1|
ISO 27002-2013 5.1.2
ISO 27002-2013 9.1.1
ISO 27002-2013 9.1.2
ISO 27002-2013 13.2.1
ISO 27002-2013 18.2.2
|AC-1 Access Control Policy and Procedures|
CM-1 Configuration Management Policy and Procedures
IA-1 Identification and Authentication Policy and Procedures
IR-1 Incident Response Policy and Procedures
MA-1 System Maintenance Policy and Procedures
RA-1 Risk Assessment Policy and Procedures
SC-1 System and Communications Protection Policy and Procedures
Depending on the software or service being built, many or all of the security-related services and corresponding security controls can be provided by underlying layers or infrastructure of the system or cloud platform.
IBM Cloud infrastructure and application platforms offer many, if not all, of the security management services listed above. Some of these services are offered as part of base functionality, and some of the services might be optional.
Modern software, systems, and services that operates on-or-near the Internet are exposed to threats and attacks that lead to security incidents, including compromised system and the loss or exposure of sensitive information.
Malicious actors and malicious software may probe the software, systems, and services to identify weaknesses and vulnerabilities in code and integration. These threats will exploit weaknesses and vulnerabilities to break the targeted software, corrupt the target systems, or steal sensitive data that may be stored in the service.
Threat modeling offers insight to the development team on actions, behaviors, or conditions that could lead to security incidents. With this insight, the development team can plan for secure design, programming, configuration, integration practices, and security testing, to avoid enabling these abuse cases.
Threat models can be simple or complex.
Simple threat model
A simple threat model is a pattern or list of common programming errors based on published information such as SANS25 and OWASP Top 10. These two patterns apply to web applications. There are other patterns that apply to Internet of Things and other non-web applications.
Software architects and engineers can use a simple threat model as a guide in coding, integration, and testing. Simple threat models may be sufficient when developing application services that will run on secure cloud infrastructure and platforms.
Complex threat model
A complex threat model involves analyzing:
- Motives and actions of bad actors
- Identifying targets of attack (applications, systems and information assets)
- Documenting actions that bad actors might take to cause those risks to be realized
This analysis is combined with known methods of attack and known software weaknesses to create guidelines for design, development, and integration. We recommend that you create complex threat models for Infrastructure as a Service (IaaS), Platform as a Service (PaaS), Cloud Operations Services, and Business Support Services (OSS and BSS).
Whether you perform a simple or complex threat model, the output of the threat modeling process needs to be applied in technical design, programming, systems integrations, and security testing. Retain your threat model documentation, as it is a particularly important input to operational security control validation and penetration testing.
Secure coding and integration
Secure coding is a development-centric practice that reduces the likelihood of security-related design weaknesses, coding defects and integration errors occurring in software, systems and services. Developers perform secure coding by following the guidance from the threat model to avoid dangerous programming, software configuration, and integration errors.
The focus of secure coding includes ensuring proper syntax, construction, integration and configuration to avoid vulnerabilities and weakness related to these and other considerations in software development:
- Input validation
- Output encoding
- Session management
- Credential and password handling
- Protection of sensitive data in storage and in motion
- Protection of log information
- Selection and proper use of APIs and network services
Security testing validates the correctness of design, coding, and integration of software, systems and services. Security testing activities can happen at any of multiple points in the DevOps phases of software and services.
Select the tool(s) and technique(s) to validate the successful completion of secure coding, configuration, and integration activities.
- Source code scan
- Local instance of AppScan Source
- IBM Application Security on Cloud Service (ASOC)
- Dynamic security scan
- Local instance of AppScan Standard
- IBM Application Security on Cloud Service (ASOC)
- Manual code review
- Code walk through
- Formal code review
- Validation testing of integrated services
- Automation-driven security tests
- Human-driven security tests
Vulnerability and incident management
Vulnerability and incident management is a practice that is shared between development and operations. Vulnerability management refers to detecting and managing vulnerabilities in deployed software and services.
Operational systems need to be scanned and tested regularly to ensure that the integrity of the software, system, or service remains intact. In addition, it is important that development personnel monitor security feeds for vendors and other public sources to learn about newly discovered vulnerabilities in components that are relevant to their software, systems, and services.
Once a vulnerability is detected, the development team must analyze the notification in a timely manner, and resolve, distribute, and deploy fixes for the security defects and vulnerabilities. User notification of vulnerabilities may be required by an organization’s policy, subscriber contract, or legal obligation.
Operational controls validation and improvement
Operational controls validation and improvement is a practice that is shared between development and operations.
Security threats and technical vulnerabilities continue to evolve. It is important to include a continuous improvement process between operations and development. Good security requires a combination of good design, good operations, good vulnerability management practices, comprehensive security testing practices, and security intelligence from external sources.
Sustainable security requires analysis of the effectiveness of security in software, systems, and services, along with the history of security events and conditions found in operations. The results of this analysis should feed the next iterations of secure development to ensure that prior security defects are not revisited, and emerging threats are addressed quickly.
Secure deployment and operations
Secure deployment ensures the application delivery processes, pipeline and the deliverables are secured and free from vulnerabilities. Secure DevOps should also ensure that the code, components, and packages are deployed securely, with the proper checks and balances.
In secure deployment, a new or updated delivery is ready for deployment and has been tested to ensure that the delivery addresses the relevant threats in the threat model.
A delivery can include new or updated functions, patches to resolve defects in existing functions, or patches to address security vulnerabilities. Depending on the type of delivery, the change to production systems and services can be normal, expedited, or an emergency.
It’s also important to note whether a change may have a security impact. You can largely automate changes that have no security impact. If a change has a security impact, you should include a level of review and approval that is consistent with the operational risk.
Secure deployment, vulnerability, and patch management must be a core part of your DevOps practices, covering:
- Cloud infrastructure
- Cloud platform
- Cloud application runtimes and services
- Cloud applications
Furthermore, it should cover both in-house software components and services, as well as supply chain components and services.
The vulnerability and patch management may be driven from any of a number of activities that detect security weaknesses in deployed systems, including:
- Penetration testing
- Network vulnerability scans
- Security compliance checks for operating system hardening, firewall configuration, and user access management
- Web application dynamic scans
- Checking that the configuration and policies are set correctly for the deployed components
- Security validation of tools and scripts
- Scans of systems to build software inventories, with versions, release and patch levels
Vulnerability and patch management for a cloud platform
Choosing a cloud platform that provides automated security vulnerability and patch management simplifies the process for your team and keeps your cloud platform secure.
IBM Bluemix Public
IBM maintains and installs updates and fixes to the IBM Bluemix® platform, runtimes, and services. Bluemix services use pre-defined, standard maintenance windows, which occasionally cause the services to be unavailable.
On the Bluemix status page IBM sends broadcast messages of the changes that are planned for each maintenance window. In addition, IBM works with you to schedule maintenance updates for the Bluemix platform. IBM uses IBM Endpoint Manager to automate patch management for Bluemix Public. This takes the guesswork out of when to conduct patch management procedures.
IBM Bluemix Dedicated
When implementing security vulnerability and patch management in your Bluemix Dedicated environment, you need to clearly define the required tasks and the owner for each task—from inception, progression, and completion. See the Bluemix documentation for information about Bluemix Dedicated roles and responsibilities.
Someone on your team needs to coordinate with IBM to schedule maintenance updates. You provide IBM with pre-approved maintenance windows and specific dates or times that might not work for you, and IBM works to schedule updates that meet your timeline.
In your Bluemix Local environment, you need to understand what roles you are responsible for and what role IBM is responsible for in terms of maintaining your Bluemix Local instance.
Refer to the Bluemix Local documentation for a specific list of roles and responsibilities. In general, IBM installs, remotely monitors, and manages Bluemix Local in your data center through Relay, a delivery capability included with Bluemix Local. Relay connects securely with certificates specific to each Bluemix Local instance. For more information about Bluemix Local and Relay, see the Bluemix Local documentation.
For SoftLayer, IBM provides the operational management of its physical facilities and the infrastructure (network and servers hosted in this environment), while you are responsible for the architecture, deployment, security, and operational management of your SoftLayer hosted workload.
The roles and responsibilities for monitoring and ensuring security of various infrastructure components are detailed in the table below.
Figure 2. Roles and responsibilities: physical, virtual network management
With the virtualization-based cloud, SoftLayer assumes responsibility for the management of the underlying (physical) host and hypervisor, including configuration and patching. This includes hardening of the hypervisor and patching, both scheduled and emergency.
SoftLayer provides local update mirrors in private networks, so servers always have the latest operating system security patches and upgrades. This allows users to initiate and install patches and updates on-demand with no-cost, unlimited bandwidth.
All update services at SoftLayer share the following features:
- Update traffic is part of your unlimited private network bandwidth and does not reduce public bandwidth allotments.
- Updates speeds are faster than updating from vendors directly.
- Updates are readily available during worldwide heavy critical update days.
- In emergency situations, public ports can be shut down while still allowing updates to occur over the private network
- Servers are pre-configured to update automatically at deployment with manual updates available on demand.
Vulnerability and patch management for cloud runtimes and services
IBM periodically assesses all its cloud services to determine security risk against security readiness criteria based on security policies. IBM assigns security readiness focal points to each cloud service to ensure the service conforms to security and compliance readiness. Operating systems and cloud runtime and middleware need to be regularly patched to guard against newly found vulnerabilities or to provide additional functionality. IBM Cloud makes the latest runtime and middleware services available through buildpack updates in Bluemix.
Vulnerability Advisor for containers
IBM cloud offerings include Vulnerability Advisor, a capability of IBM Containers to discover vulnerabilities and compliance policy problems in Docker images hosted in Bluemix. Using Vulnerability Advisor, developers can design secure applications with very little effort.
The Vulnerability Advisor integrates with the DevOps lifecycle, inspecting Docker containers without requiring any agents.
When you add an image to your private Bluemix repository, each of the packages in the image are compared automatically by the Vulnerability Advisor against policies set by the organization manager and a database of known issues.
You can review the vulnerability manager results (pass or fail status) for all images. The image’s vulnerability report flags:
- Vulnerable packages: Details of all vulnerable packages and security violations with specific information about the vulnerable package and steps to update the package.
- Audit violations: List of suggested compliance checks to increase the security of the image.
The Vulnerability Advisor Dashboard provides an overview of an organization’s images, vulnerability settings, and vulnerability assessment. If you are the manager of your organization, you can set policies that either warn users or block deployment of images with specified vulnerabilities.
Learn more about Vulnerability Advisor and reviewing potential vulnerabilities in an image from the Bluemix Dashboard.
Vulnerability and patch management for cloud applications
Vulnerabilities in a cloud application pose big risks to customers. Threat modeling to identify relevant weaknesses and attack patterns related to the application and secure design helps mitigate these risks. Secure coding guides and practices prevent vulnerabilities. Security testing fixes problems before the app is deployed and validates that the product is free of known security issues.
IBM Application Security Service on Cloud helps secure your organization’s applications by detecting dozens of today’s most pervasive published security vulnerabilities. As such, it helps to eliminate vulnerabilities from applications before they are placed into production and deployed.
Through convenient, detailed reporting, the IBM Application Security Service helps you:
- Identify vulnerabilities before release. The service scans applications at the appropriate stage of your development lifecycle.
- Identify security vulnerabilities to real threats, permitting you to focus on vulnerabilities that are most likely to have a significant impact on your organization.
- Provide a quick solution with built-in recommendations for how you can remedy security issues for your apps.
Vulnerability and patch management for the supply chain
Security controls within the supply chain of people, technology, and services is critical to developing and delivering a secure cloud and ensure the third-party components are free from vulnerabilities. IBM has a mature practice for vetting the open-source and third-party software used in our products and services.
The Open Source Council at IBM governs open source adoption within IBM. The process includes nomination and review of open source offerings, scanning open source offerings for licensing and security, cataloging and tracking the components within open source packages, as well as tracking where the packages are integrated into software products and offerings.
Separation of duties
Separation of duties ensures users have only the access that is required to perform their jobs. You must ensure that there is a separation of roles and responsibilities so that only users with the correct roles are given access to specific areas of your cloud development and deployment process.
Within Bluemix, we follow Separation of Duties guidelines to assign granular access privileges to users, and to ensure that users have only the access that is required to perform their jobs according to the principle of least privilege.
Within a Bluemix Dedicated and Local environments, assigned administrators can manage roles and permissions for Bluemix user in their organization by using the Admin Console. See Managing Bluemix Local and Dedicated for details.
Availability and business continuity management
Availability and business continuity management ensures your infrastructure, runtime components, and management components are highly available.
Bluemix Public provides a continuously available platform for innovation with multiple fail-safe measures to ensure that your orgs, spaces, and apps are always available. Deploying apps to multiple geographic regions enables continuous availability that protects against unplanned, simultaneous loss of multiple hardware or software components, or the loss of an entire data center. Because of this, even in the event of a natural disaster in one geographic location, your distributed Bluemix Public app instances in alternate geographic locations will be available.
Bluemix separates those components that track the state of interactions (stateful) from those that do not (stateless). This separation allows Bluemix to move apps flexibly as needed to achieve scalability and resiliency.
Learn more about Bluemix Public resiliency.
Disaster recovery for Bluemix Dedicated can be set up similarly to the way that it works when you use Bluemix Public. Disaster recovery for Bluemix Dedicated is made possible through continuous availability for your apps, the inherent high availability of the platform, and the ability to restore your instance in the event of a failure.
Learn more about Bluemix Dedicated disaster recovery and how to deploy Bluemix Dedicated apps to multiple geographic locations.
Disaster recovery for Bluemix Local can be set up similarly to the way that it works when you use Bluemix Public. Disaster recovery for Bluemix Local is made possible through continuous availability for your apps, the inherent high availability of the platform, and the ability to restore your instance in the event of a failure.
You are responsible for enabling continuous availability of your apps by deploying to multiple regions. Learn more about enabling continuous availability for Bluemix Local.
SoftLayer provides a highly available environment that IBM manages for availability, disaster recovery and business continuity. You are responsible for using the SoftLayer environment to ensure that your deployed environment meets your requirements for high availability (HA), disaster recovery (DR), and business continuity (BC). Typically this implies that you have deployed your workload across at least two pods and ideally across two geographically distributed data centers.
Because of SoftLayer’s extensive global presence, it’s possible to establish a two data center approach to high availability, disaster recovery, and business continuity in most geographies including those with strong data locality requirements.
Learn more about IBM SoftLayer management.
Security evaluation and learning
Security evaluation and learning practices make sure that the security functions and properties in the delivered code and services are maintained as threats evolve and new vulnerabilities arise.
Security evaluation and learning occurs on multiple levels. When you choose to procure, install, configure, operate, and maintain computing systems in your enterprise, you need to ensure that the development, operations, and support teams have full visibility into the security issues at all layers of the technology stack. These teams must be able to affect the changes needed to adapt the security of software, systems, and services to new threats and newly discovered vulnerabilities.
IBM X-Force Managed Security Services can help your enterprise and your teams evaluate security and learn from security events.
IBM tracks its cloud users’ security and service continuity concerns and addresses them within our infrastructure, platforms, and services, through our use of IBM X-Force as well as our Security Operations Centers and personnel.
You still need to have visibility into security issues for your applications deployed in the IBM Cloud, your data stored in the IBM Cloud, your business continuity concerns, as well as the administration of security services for your employees and your end-users, such as identity, and access management.