Taxonomy Icon



In current computing environments, applications rely on system software for providing services, such as managing access to the computing system’s resources. System software (at the minimum) includes an operating system (OS) but may also include a hypervisor. Traditionally, system software must be trusted, because it has complete control over applications and their data. The operating system or the hypervisor can access or modify the data of any application, or potentially tamper with any security features implemented by the application without being detected. Consequently, the underlying software must be part of the Trusted Computing Base (TCB). In shared environments, customers are forced to trust that the entities that develop, configure, deploy, and control the system software are not malicious. Customers must also trust that the systems software is invulnerable to attacks that escalate privilege and compromise the confidentiality and integrity of the customers’ applications. This broad trust requirement is often difficult to justify and poses a significant risk, especially for customers who adopt public cloud services. This article describes a new approach 1 introduced by IBM® to address such concerns. Note that some features might be different when the product is introduced.

Addressing these concerns for security-sensitive applications is part of the motivation for technologies such as IBM SecureBlue++ 2 , 3, Intel SGX 4 , 5 , 6, and AMD Secure Encrypted Virtualization 7 , 8. IBM is enabling support for secure virtual machines (SVMs) on IBM Power Systems™. The IBM Protected Execution Facility protects SVMs from other software while the SVMs are at rest, in-transit, and while running. To support these SVMs, IBM is introducing a new mode in IBM Power Architecture®, called Ultravisor mode, represented by the red box in Figure 1, that has higher privilege than the hypervisor mode.

Figure 1. Modes in the new system architecture

As shown in Figure 1, privilege decreases as you move from the highest privilege Ultravisor mode, to the lowest, Problem mode.

An SVM can run only on systems that support the IBM Protected Execution Facility and are designated by the customer who created the SVM. Each system that supports the IBM Protected Execution Facility has a public/private key pair where the private key is known only to the system (it is not exposed to the owner of the system). Using tooling to create the SVM, the creator of an SVM supplies the public keys for each system that is authorized to run the SVM. Initially, at the time of manufacture, IBM causes a unique system identification key pair to be generated on each system and delivers the corresponding public key to the owner. Upon delivery, the system owners can change their key pairs using the tooling supplied by IBM. Using the facilities of the Trusted Platform Module (TPM), system owners might choose to use different keys for each system or the same key for a group of systems. Note that IBM is not involved in authorizing SVMs.

To create an SVM, the customer must:

  • Create a VM that will become the SVM.
  • Acquire from the system owners the public key of each system on which the SVM is authorized to run.
  • Utilize the tooling supplied by IBM to build the SVM from the VM.
  • Distribute the SVM to one or more of the systems where the SVM is to run.

After the SVM is created (including any sensitive information such as keys), it is cryptographically protected at rest and throughout execution.

Figure 2 shows an overview of the Protected Execution Facility. The bottom section of the picture represents the hardware environment, the middle section represents the firmware/software environment, and the top section represents the application environment. The Protected Execution Facility is built upon secure and trusted boot. Each system has a public/private key pair where the private key is protected by a TPM and is useable only if the correct, verified, firmware has been launched. The lower-right side of the figure shows that memory is partitioned into secure memory and normal memory. As part of the secure boot process, the Protected Execution Ultravisor (or Ultravisor) is loaded into secure memory. After this firmware has taken control of the system, it passes control to the OS, usually a hypervisor, that starts running. As indicated at the top portion of the figure, the public key of the target system is used to wrap the encryption key of the secure VM when it is created. When the SVM starts, the software in the SVM calls the Ultravisor which moves the SVM to secure memory where execution resumes. Normal VMs can run on the same hardware, but without any additional protection.

Figure 2. Overview of the Protected Execution Facility

An SVM uses hypervisor calls (h-calls) to obtain services from the underlying hypervisor. Our model introduces a new mode called Ultravisor mode with associated firmware and ultravisor calls (u-calls). This firmware, which we refer to as the Ultravisor, will be open sourced to provide increased transparency and to allow the community to review and strengthen its security. The Ultravisor filters all calls between an SVM and the hypervisor to assure that only the information required to perform the call is passed to the hypervisor. It also makes sure that only the response from the hypervisor goes back to the SVM. The hardware prevents unauthorized system software (and hardware), for example the hypervisor and normal virtual machines, from referencing secure memory. However, the hypervisor retains all its traditional functions.

It is important to note that part of the protection for an SVM at rest is that its disk is protected with disk encryption, such as Linux dm-crypt. This description raises a few obvious questions:

  • With the hypervisor maintaining its normal function, but prevented from referencing secure memory, how are SVMs started?
  • How does paging work? Does it give the hypervisor access to the SVM memory?
  • How does I/O work?
  • This is a significant change; how will it impact normal VMs?
  • What is the performance impact?
  • Exactly what does a protected SVM disk image look like?

We address these questions in the rest of this article.

Starting an SVM

Because a SVM is formatted like a zImage, existing boot loaders should be able to load it without modification. When the hypervisor starts a VM, it copies the VM (or part of the VM) into memory and transfers control to the entry point indicated in the VM. An SVM has a similar format except that the entry point of an SVM is in software that is integrity-protected and unencrypted. One of the things this software must do is transition from running as a normal VM to running as a SVM. This is done by making an Enter Secure Mode (ESM) u-call. In response to this call, the Protected Execution Ultravisor:

  • Copies the SVM to secure memory.
  • Checks if the system is authorized to run the SVM.
  • Verifies that the clear text has not been modified (if the system is authorized).
  • Decrypts (with integrity checking) those parts that need decryption. If the integrity checks pass, execution resumes in secure memory.

After the SVM has been copied to secure memory, execution resumes in secure memory. One feature of secure memory is that it cannot be addressed by the hypervisor.


If paging is required, the Ultravisor makes an encrypted version of the requested SVM memory page available to the hypervisor. (Our architecture supports memory overcommit for SVMs, although support for overcommit may not be available initially.) The cryptographic mode used to protect the SVMs is an extension of the Advanced Encryption Standard (AES) called Integrity Aware Parallelizable Mode (IAPM) 9 , 10. IAPM detects any modifications of the encrypted representation of an SVM page by the hypervisor (or anyone else). Similar capabilities were initially demonstrated in a research project 11.


The SVM can specify to the Ultravisor that specific pages of its memory are to be shared with the hypervisor without protection. This causes the page table for the SVM to point to pages in normal memory. Initially all I/O in the SVM will be restricted to virtual I/O. Because the hypervisor can only reference pages that have been explicitly requested to be shared with it by the SVM, VirtIO must be modified to work in an SVM. Bounce buffers must be used because the hypervisor cannot observe anything inside the SVM. The SVM is responsible for properly protecting (encrypting) any data that is written through shared pages. Shared pages are necessary so that the SVM can communicate with other entities because the keys that protect its memory are unknown outside the system it is running on. If the creator of the SVM is concerned about the security of any communication, Transport Layer Security (TLS) or a similar mechanism should be used to protect the data. For a communications protocol, TLS inside the SVM encrypts the data in secure memory before it is placed in unprotected memory.

When the SVM’s disk is protected with an encrypted file system, such as dm-crypt, the passphrase must be made available to the kernel as the VM boots without console interaction. An SVM might have additional disks protected with dm-crypt (or a similar feature), where the passphrases for these disks are protected inside the SVM.

Dedicating devices to a VM for I/O

Initially, we might not allow dedicated devices to be assigned to an SVM.

Performance impact

There is only minimal (essentially zero) impact on normal VMs. The impact on SVM is minimized because the interface between the hypervisor and the Ultravisor is paravirtualized. Some facilities that were previously controlled by the hypervisor are now controlled by the Ultravisor. When the hypervisor needs to use these facilities, its actions must be acceptable to the Ultravisor.

Non-protected VM disks

A disk that is not protected by dm-crypt or a similar mechanism can be used by an SVM. However, unless the data is encrypted within the SVM by another mechanism, there will be no protection for data written to or read from that disk.

Protected SVM disk image

If the SVM is loaded by petitboot 12, the contents of the SVM would look similar to the one shown in Figure 3. The first part contains the loader and additional information as indicated. The loader does the ESM u-call. The boot strapping code does the ESM u-call. This call returns to the SVM only if it is authorized to run on the current system and all the areas verifiable by the Ultravisor have been checked. The next area indicates the amount of shared memory that will be used for services during the boot process. This is followed by the ESM blob which is self-protecting and contains the symmetric key encrypted under the public key of the target system and other meta data used for managing the SVM. Finally, there is the boot file system and the root file system. Digital signature protection means that before running, the signature on the kernel of the boot file system and the signature of all files run from the RAM disk must be checked. If any signature check fails, the boot fails. Block layer encryption refers to a system such as dm-crypt.

Figure 3. Format of an SVM image when based on petitboot


We have built a facility that allows protected execution of SVMs on IBM Power Architecture. Because SVMs are encrypted, these virtual machines are protected when on disk or being transmitted over a network (even if encrypted transmission is not used). Each system is configured with a public/private key pair. The information authorizing the SVM to run is encrypted under the public key of an authorized system. An SVM can be authorized to run on multiple systems. As the SVM is being loaded, the authorized system verifies that it has not been tampered. If a tamper is found it will not be executed. The SVM uses hypervisor services but is protected from inadvertent leaking of information to the hypervisor.


The authors would like to thank the following IBM colleagues for helpful discussions and technical review: Eric Hall, Jens Leenstra, Ramachandra Pai, Ray Valdez, George Wilson, and Sarah Wright.