In this article, we provide a brief introduction to hardware security modules (HSMs), explain why you might want to leverage them in your IBM Blockchain Platform network, and also describe the high-level activities for providing a high-availability solution around your HSM appliances. In addition, we share an alternative to HSM appliances for securing digital keys in your IBM Blockchain Platform network.
Though providing step-by-step instructions on how to configure your HSM appliances for high availability in your IBM Blockchain Platform network is beyond the scope of this article, we include direct links and pointers to the documentation that provides that level of detail.
Overview of HSMs
An HSM is a physical appliance for managing and securely storing digital keys of high value. In addition to safeguarding these keys, HSMs perform cryptographic functions with those digital keys, such as signing transactions.
As its name implies, an HSM is built with security in mind first. A common feature in many HSM devices is being tamper resistant: If tampering is attempted, the HSM appliance can become inoperable or digital keys in it can be automatically deleted. In many business situations, if such keys were compromised, there could be a huge negative business impact to the organization or business that owns them. A leak of private keys could grant access to privileged information hosted in the world state and ledger. It could even enable someone to issue transactions on the network.
Because HSMs play a critical role in securing applications and are typically part of systems that are essential for the operations of a business, many HSM manufacturers provide capabilities for clustering the HSM appliances for high availability and performance (as this article shows). HSM appliances are usually attached to network servers and are common in payment, financial, and banking applications.
How HSMs work
Is an HSM just like a database of private keys? Not really. The real power of HSMs is that the cryptographic material is generated in the HSM and never gets out of the device. As mentioned in the previous section, the HSM provides cryptographic functions that allow authorized systems to invoke functions (like signing and encryption), which leverage the private key stored within the device. You send your data to be signed and encrypted to the HSM device, and therefore, you don’t need to retrieve the key.
For example, the following diagram depicts the process for requesting a key pair (in other words, public and private keys) to an HSM. Notice that the Certificate Signing Request (CSR) is sent to the HSM for signature. The private key ID (referred to as handle in the diagram) is used to access the private key.
While the CSR is an important concept in the public key infrastructure (PKI) world, the HSM sees a CSR as just another data structure to sign. An HSM has no concepts of certificates and the other associated structures.
Leverage an HSM in your IBM Blockchain Platform network
Depending on regulatory, non-functional requirements, an organization might be required to store keys in HSM appliances.
One of the main goals of using blockchain technology is to provide irrefutable proof that a set of transactions have occurred between members of a business network. In an enterprise blockchain business network, transactions endorsements are signed, identities for the different nodes (for example peers, borderers, and more) are signed by trusted certificate authorities, channel updates must be signed by the required approvers, and orderer nodes sign blocks to be added to the ledger among other tasks. For all these digital signatures, digitals keys are used. Therefore, using an HSM appliance is a great way for safeguarding such keys and for executing any signature operations as well. When using HSM appliances for storing such keys, unauthorized access to the keys that your Hyperledger Fabric nodes use for signing transactions and endorsements becomes extremely challenging and difficult.
It is worth noting that HSM appliances are not cheap. Organizations that leverage HSM appliances do incur high costs. In the end, your organization should weigh the benefits and risks of all available options when deciding whether or not to use HSM appliances.
IBM Cloud HSM offering
With the IBM Cloud HSM offering, you can provision two types of HSM devices:
- HSM 7.0, FIPS 140-2 Level 3
- HSM 6.0, FIPS 140-2 Level 2
The manufacturer of both of these HSM devices is Gemalto, and the model for both is SafeNet Luna SA. From these two options, we recommend using HSM version 7.0 because it is the latest version of the appliance from the manufacturer and meets a higher level of FIPS 140-2 compliance.
Note that IBM Cloud HSM is an infrastructure as a service (IaaS) offering. This implies that being familiar with the operation and administration of the HSM device is required to take full advantage of this cloud offering.
Overview of HSM HA groups and why you need them
Configuring Hyperledger Fabric nodes in your IBM Blockchain Platform network to leverage an HSM appliance results in the following:
- The private identity key of the Hyperledger Fabric node is stored in the HSM appliance.
- Cryptographic operations that use that private key, such as signing transactions, occur within the HSM appliance.
Given this, it is important to consider what happens if the HSM appliance stops working (in other words, malfunctions) or the data center where the HSM appliance is found becomes nonoperational. In either scenario, you have an outage, which might be unacceptable for your business. What, then, are the available options to prevent this type of outage?
To address high availability (HA) and disaster recovery (DR) concerns, the organization that has full administrative rights over the HSM appliance should create an HA HSM group. To do so, you need multiple HSM appliances. A recommended configuration for an HA HSM group is to have three (3) HSM appliances, where two are active and one is in standby mode. In this configuration, cryptographic requests are load balanced between the two active appliances, while the third device sits on the side and only becomes active whenever one of the two active appliances fails to function. The HA HSM configuration ensures that all cryptographic objects (for example, keys) are replicated across the HSM appliances that are part of the same HA group, even to those HSMs that are in standby mode. Therefore, an HA HSM group gives you the capability to have current, automatic backups of the cryptographic contents in your HSM appliances (which is a must for addressing DR concerns).
You can think of an HA HSM group as a virtual logical group that is composed of the physical HSM appliances that you administer (note that provisioning an HSM device through the IBM Cloud HSM offering grants you full administrative rights over that appliance).
For example, say you have an IBM Blockchain Platform network deployed to a three-zone IBM Kubernetes Service (IKS) cluster on the IBM Cloud. You can then have one HSM device provisioned in each zone (or data center) of your IKS cluster, for a total of three HSM appliances. Finally, you can create an HA HSM group that is composed of these three HSM appliances, where two HSMs are active while one is in standby mode.
IBM Blockchain Platform integration to HSMs
The PKCS#11 application programming interface is a standard in the world of PKI devices and enables consuming applications to communicate with HSM appliances such as the SafeNet Luna devices. All serious HSM vendors provide PKCS#11 libraries.
Clients of the HSM are not always located on the same process namespace or network location as the HSM server. In such cases, it might be good to have a PKCS#11 proxy, which can serve as the bridge between the client application and the library provided by the HSM manufacturer.
In the context of the IBM Blockchain Platform, it relies on a PKCS#11 proxy service, which embeds the HSM appliance library (in our case, the SafeNet Luna devices). The proxy is used as a bridge between Fabric nodes and the library provided by the HSM manufacturer. We like to refer to this software component as the HSM proxy.
To reiterate, the HSM proxy component 1) uses dynamic linking for loading at runtime the library provided by the HSM manufacturer and 2) provides a TCP endpoint for communication, which allows decoupling from the Hyperledger Fabric nodes (in other words, the HSM proxy component does not need to be part of the container image that hosts the Hyperledger Fabric node). You can think of the HSM proxy as a pass-through component between the Fabric nodes in your IBM Blockchain Platform network and your HSM appliance(s).
A common embodiment for the HSM proxy is a Kubernetes pod (for instructions on how to create this HSM proxy pod component, see IBM Cloud Hardware Security Module (HSM)). Just like you need multiple HSM appliances to achieve HA at the infrastructure level, you also need multiple instances of the HSM proxy pod running in your Kubernetes cluster to achieve HA. For example, if you only have an HSM proxy pod and the zone where that pod is running has an outage, then even with an HA HSM group in place, you will have downtime.
To minimize the situation where there aren’t any HSM proxy pods running, you can create a replica of the HSM proxy pod — one pod per zone in your IKS cluster. This ensures that if one of the pods goes down (or if the entire zone/data center where one of the HSM proxy pods resides goes down), your IBM Blockchain Platform network is still up and running. Please note that, by default, all HSM proxy pods/replicas should be automatically spread across the different zones in the IKS cluster.
For the SafeNet Luna devices, creating a replica of an HSM proxy pod requires performing the following two tasks upfront:
- Configure the HSM appliances (initializing the HSM device, creating partitions, defining passwords, and so on)
- Define the HA HSM group
Completing these two tasks upfront means executing them prior to the deployment of the HSM proxy pods. For instance, you can launch a VM or a Kubernetes pod, where its only purpose is initializing the HSM appliances and defining the HA HSM group. Once these tasks are completed, you will have:
- The HSM client configuration file (Chrystoki.conf) with the necessary metadata to establish connectivity to the HSM appliances along the HA HSM group definition
- The necessary PKI certificates and private keys (see Configuration Guide) required for obtaining access into the HSM appliances
Finally, you can then have your HSM proxy pods leverage the HSM configuration file and PKI files by using Kubernetes ConfigMaps and Secrets. The following diagram depicts the configuration we have described here for HSM appliances and HSM proxy pods:
Leveraging an HA HSM group from your IBM Blockchain Platform network
To leverage an HA HSM group from the IBM Blockchain Platform Console, you simply need to provide the following parameters when provisioning your Hyperledger Fabric node, such as a Hyperledger Fabric certificate authority node:
- HSM proxy endpoint: TCP endpoint to access the HSM proxy over the network
- HSM label: The name of the HA HSM group (an HA HSM group is a virtual partition)
- HSM PIN: The password for the partitions that make up the HA HSM group
As you can tell from these parameters, the IBM Blockchain Platform cannot tell whether the parameters you provide to it correspond to an HA HSM group or to a single HSM physical partition.
Alternative to HSMs
The IBM Blockchain Platform stores, by default, the private keys associated with the identities of your Hyperledger Fabric nodes (certificate authorities, peers, and orderers) in Kubernetes secrets. A Kubernetes secret is an object meant to store sensitive data. Storing sensitive data in secrets gives you control over access and usage of such data. By leveraging Kubernetes secrets, you keep sensitive data outside your Kubernetes pods and containers, which frees up developers from having to bundle the application code with restricted information. System and infrastructure administrators can then provide critical and sensitive data (such as passwords, private keys, and more) to applications during deployment and/or runtime.
Because the data in Kubernetes secrets is stored in base64 encoding, you might need additional security measures for your Kubernetes secrets data. For instance, you can add another layer of security by encrypting your Kubernetes secrets data at rest. On the IBM Cloud, you can do so by:
- Provisioning an instance of the Key Protect service
- Importing your own symmetric keys for encryption into Key Protect (or you can generate new symmetric keys using Key Protect)
- Configuring your IBM Kubernetes Service cluster instance (where the IBM Blockchain Platform will be deployed to) to use Key Protect as the Key Management Service (KMS) provider
When you take these steps, your Kubernetes secrets are encrypted with a symmetric key stored in Key Protect. Key Protect uses a FIPS 140-2 Level 2 certified cloud-based HSM appliance for storing your symmetric key(s). However, note that this approach is by no means equivalent to using your own HSM device for storing the keys associated with your Hyperledger Fabric nodes. Instead, Key Protect uses shared HSM devices (which they solely administer) to store the symmetric key for encrypting your Kubernetes secrets data at rest. It is a subtle and relevant difference to keep in mind.
Though simply leveraging Kubernetes secrets or also adding Key Protect as a KMS provider to your IKS cluster are valid options for storing private keys, these are not equivalent to using an HSM appliance. As described previously, HMS appliances provide higher security for safeguarding private keys. For instance, an HSM appliance might go out of service the moment tampering is detected. Also, cryptographic material generated in an HSM appliance does not leave the device, and, as a result, cryptographic operations occur within the device itself.
To sum up, you can leverage Key Protect to encrypt the sensitive data stored in your Kubernetes secrets as opposed to just simply encoding that data in base64 format. For more details on leveraging Key Protect as the Key Management Service for your IKS cluster, see IBM Key Protect is Now Available for IBM Cloud Kubernetes Service.
Outages and data loss in your IBM Blockchain Platform network could have a negative impact on your business, such as financial losses and/or loss of trust from customers and other members of your blockchain network consortium. Having a sound HA architecture helps mitigate the possibility of outages and downtime in your blockchain network and helps your organization meet non-functional requirements around reliability and redundancy.
Ensuring the deployment of your HSM components (like the HSM appliances and HSM proxy pods) follows best practices for high availability and recovery from data loss should be part of your overall architecture and design efforts for your IBM Blockchain Platform network.
It is worth mentioning that in addition to storing the keys for the Fabric nodes (in other words, peers, orderers, and certificate authorities) in an HSM appliance, you should also consider storing the keys for the following in the HSM:
- Fabric client applications
- Organization’s administrative identities
Taking these measures for safeguarding these peripheral keys should provide further security. Doing so significantly reduces the possibility of compromising those adjacent software components that have access to the blockchain network. Of course, for this to be effective, you should ensure the maximum enrollments property of the identity is set appropriately at registration time (the maximum enrollments value limits the number of times that the same password can be used for generating a private key and corresponding certificate). For further details on the maximum enrollments property, see the Fabric CA documentation.
If leveraging HSM appliances is not a feasible option for your organization but you would still like to provide an additional layer of security for your keys, you can instead configure your IBM Kubernetes Service cluster instance to use Key Protect as the KMS. This action ensures that your digital keys for the Fabric nodes are encrypted at rest.
To learn more about PKCS #11, visit this great resource from Red Hat. If you are ready to provision an IBM Cloud HSM appliance for your IBM Blockchain Platform network, visit the Cloud HSM page on the IBM Cloud catalog. Finally, don’t forget to visit the IBM Cloud Hardware Security Module (HSM) page under the IBM Blockchain Platform documentation.