Build inheritance trees for virtual machines and templates

Virtual machine and virtual machine templates are widely used in the virtualization world today. Building virtual machine templates even becomes the daily work of their providers as the change requests keep coming more frequently than ever, either from new features or security enhancements.

Building a template is usually an iterative process with the loop of developing, debugging, and testing, which includes such activities as deploying a virtual machine from an existing template, making changes on the virtual machine, capturing the virtual machine to a new template, as well as backing up current virtual machines and templates, as shown below. During the process, it is common to see quite a few virtual machines and templates existing in the development environment after a few iterations. The following chart also gives an example of the vCenter view in a real templates development environment.

It would be helpful for people who are building the templates to understand the relationships among those virtual machines and templates — for instance, which virtual machine is deployed from which template, which template is captured from which virtual machine; in other words: the inheritance relationship between virtual machines and templates. Unfortunately, vCenter, the most popular virtual machine platform today does not provide such view of the inheritance relationship. All virtual machines and templates are organized in a flat structure on vCenter without a hint to show the relationships of virtual machines and templates.

This tutorial shows you how to track the relationship of templates and virtual machines when they are being deployed, exported and cloned, and how to build a tool for generating the inheritance tree from the virtual machines and templates in a vCenter environment.

IBM Cloud for VMware Solutions is among the world’s largest platforms that host entire VMware environments in the IBM Cloud. This tutorial also shows how to access to your vCenter Server instances in IBM Cloud and build the inheritance trees of the virtual machines and templates in your vCenter Server instances.

Learning objectives

• Understand vCenter attributes
• Use Python SDK for the VMware vSphere API (pyVmomi) to clone virtual machines and templates with custom attributes added
• Generate and display the inheritance tree of virtual machines and templates
• Use IBM Cloud for VMWare Solutions REST API to obtain the information of your vCenter Server instances

Prerequisites

• A ready-to-use vCenter Server instance from IBM Cloud for VMWare Solutions
• A valid IBM Cloud API key associated with your IBM account for IBM Cloud for VMWare Solutions
• A machine that can connect to the vCenter server of the vCenter server instance
• Python3
• Third-party Python packages: pyVmomi and pptree

Estimated time

Completing this tutorial should take about 30 minutes.

Steps

1. Build the Python tool to track relationships and generate inheritance trees

In the vCenter Web Client, objects such as hosts, virtual machines, templates, clusters, and networks can be attached with the special attributes, called Custom Attributes. Custom Attributes can carry user-specific values for those vCenter objects. Here is an example of the Custom Attributes of one virtual machine in vCenter view.

Custom Attributes will be used here to record the information of the source templates or virtual machines deployed or exported from for the new virtual machine or templates. That information is attached to each virtual machine or template for tracking its parent. By collecting the parent information of all virtual machines and templates, an inheritance tree can be generated.

The tool implemented here is written in Python. It uses vSphere Python SDK pyVmomi module to manage customer attributes for tracking parent information and uses a pptree module to display the generated inheritance tree of virtual machines and templates.

Clone virtual machine and template

In vCenter, the following scenarios are all underpinned by the Clone operation:

• Deploy a virtual machine from a template
• Clone a virtual machine from a virtual machine
• Clone a template from a template

The pyVmomi module provides the Clone method to support the cloning operation. Here is the code snippet to show the usage of the Clone method.

    vm_folder = datacenter.vmFolder  # specify the folder for the target object

    relospec = vim.vm.RelocateSpec()
    relospec.datastore = datastore
    relospec.pool = cluster.resourcePool

    clonespec = vim.vm.CloneSpec()
    clonespec.location = relospec # specify the location of target object
    clonespec.template = create_template # specify the target object is template

    task = src_object.Clone(folder=vm_folder, name=new_name, spec=clonespec)

    while task.info.state == vim.TaskInfo.State.running:
        time.sleep(2)
    if task.info.state != vim.TaskInfo.State.success:
        raise Exception('Clone task failed')

• The Clone method is applied to virtual machine and template. The method is issued by the object of the source virtual machine or template to be cloned from. A task is created when the method is invoked successfully.
• CloneSpec is required by the Clone method to provide the spec of a new virtual machine or template, including datastore, location, etc. The type of virtual machine or template can be specified in CloneSpec as well.
• More customization for the target virtual machine or template can be found in the definition of CustomizationSpec.

Add parent information to the cloned virtual machine or template

Two Custom Attributes are added and attached to each virtual machine or template for tracking the inherence relationship:

1. UUID indicates the unique identifier of a virtual machine or template. When a new virtual machine or template is cloned, a new UUID is generated and assigned to the custom attribute UUID of the virtual machine or template. The reason for UUID is that although each virtual machine or template is required to give a name, different virtual machines or templates could have the same name if they are not in the same folder, therefore, the name does not guarantee the uniqueness.
2. PARENT_UUID indicates the UUID of the template or virtual machine that the current virtual machine or template was cloned from. The value of UUID from the source template or virtual machine will be assigned to PARENT_UUID on the cloned template or virtual machine. The original template or virtual machine imported from OVA or deployed from vCenter template or ISO will have the null value of PARENT_UUID.

Custom Attributes are managed by vSphere object CustomFieldsManager and defined by CustomFieldsManager.FieldDef. Custom Attributes support managed objects in vCenter, including virtual machine, host, cluster, resource pool, etc., but are defined with different types. The type of vim.VirtualMachine is used for the Custom Attributes UUID and PARENT_UUID as they are attached to either virtual machines or templates.

Here is how to define a new Custom Attribute:

customFieldsManager.AddFieldDefinition(name=attr_name, moType=vim.VirtualMachine)

This method creates a CustomFieldsManager.FieldDef instance by giving a name and a type. The CustomFieldsManager.FieldDef instance is at vCenter level. Once it is created successfully, the Custom Attribute is added to all existing virtual machines and templates in vCenter, and will be added to any new virtual machines and templates as well. By default, its value is None. Following is the example of definition of UUID and PARENT_UUID created by the above method.

(vim.CustomFieldsManager.FieldDef) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   key = 102,
   name = 'UUID',
   type = str,
   managedObjectType = vim.VirtualMachine,
   fieldDefPrivileges = <unset>,
   fieldInstancePrivileges = <unset>
}
(vim.CustomFieldsManager.FieldDef) {
   dynamicType = <unset>,
   dynamicProperty = (vmodl.DynamicProperty) [],
   key = 103,
   name = 'PARENT_UUID',
   type = str,
   managedObjectType = vim.VirtualMachine,
   fieldDefPrivileges = <unset>,
   fieldInstancePrivileges = <unset>
}

Internally, CustomFieldsManager assigns a unique key to each Custom Attribute when created. The key will be used as the only identifier of the Custom Attribute by other methods of CustomFieldsManager.

Here is how to set value to the Custom Attribute on a virtual machine or template:

customFieldsManager.SetField(entity=vm, key=attr_key, value=attr_value)

The value of Custom Attribute can be read from the field customValue of virtual machine or template object. Here is an example of customValue from a virtual machine:

(vim.CustomFieldsManager.Value) [
   (vim.CustomFieldsManager.StringValue) {
      dynamicType = <unset>,
      dynamicProperty = (vmodl.DynamicProperty) [],
      key = 102,
      value = 'ea3bf8a2-f098-40c9-b973-66eefe9c1d08'
   }
]

It is worth noting that the value is associated with the key of Custom Attribute instead of the name, and as a result, should be obtained by matching the key of Custom Attribute.

for customer in vm.customValue:
    if customer.key == attr_key:
        return customer.value

Generate virtual machine and template inheritance trees

A virtual machine or template can only be cloned from one virtual machine or template, which means a cloned virtual machine or template can only have a single parent. If using nodes to represent virtual machines or templates, and using edges represent the relationship of parent and child, the cloned virtual machines and templates and the inheritance relationships among them form a set of trees. The root of each tree is the original virtual machine or template that is not cloned but deployed from ISO or imported from OVA. The number of trees equals to the number of original virtual machines and templates. Each node could have more than one child that depends on how many times the parent virtual machine or template has been cloned.

A Python dictionary is used to record all nodes of virtual machine and template nodes, indexed by UUID as keys. The values of the dictionary have the other information of the virtual machine or template, including PARENT_UUID. The dictionary is generated and filled in by iterating all vim.VirtualMachine objects in vCenter. The UUID and PARENT_UUID will be added into the dictionary. Here is an example of the dictionary with two elements: a parent template and a child virtual machine:

{
    "550c451a-d697-474e-a02f-1f49d6980fb0": {
        "parent_uuid": "1e9ffe3e-8ac4-43f9-bf71-62b6a8511018",
        "name": "74rhel-vm-2020127155721-src",
        "type": "VirtualMachine"
    },
    "1e9ffe3e-8ac4-43f9-bf71-62b6a8511018": {
       "parent_uuid": None,
       "name": "rhel74",
       "type": "Template"
    }
}

The tool uses Python package pptree to print a tree-style diagram in the console. Node is the basic and only class provided in the pptree package, and it is instantiated by feeding the name and parent Node object, as shown below. The method print_tree from the pptree package does a pretty print; when a root Node object is given, it prints out the whole tree:

parent_vm = Node(name='parent VM', parent=None)
child_vm = Node(name='child VM', parent=parent_vm)

Two things need to be implemented when using pptree:

1. Convert the dictionary of virtual machines and templates into a group of pptree Node objects.
2. Find the root Node objects and call print_tree for each one.

The conversion algorithm is a depth-first traversal on the dictionary. During iterating the elements of the dictionary, always attempt to locate the parent element by checking the PARENT_UUID until PARENT_UUID is None. Meanwhile, create a new pptree Node object when hitting each element if the Node object is not created on it yet. All Node objects created on the elements with PARENT_UUID value of None are put in a separate list for the final use of calling print_tree. Here is the complete tool code:

# file name: vm_relation_tool.py

import os
import uuid
import ssl
import time
import atexit
import argparse

from pyVmomi import vim
from pyVim.connect import SmartConnect, Disconnect

from pptree import Node, print_tree


def get_smart_connetion(vcenter_host, vcenter_user, vcenter_password):
    context = ssl.SSLContext(ssl.PROTOCOL_SSLv23)
    context.verify_mode = ssl.CERT_NONE
    sc = SmartConnect(host=vcenter_host, user=vcenter_user, pwd=vcenter_password, port=443, sslContext=context)
    atexit.register(Disconnect, sc)
    return sc


def get_obj(content, vimtype, name):
    container = content.viewManager.CreateContainerView(content.rootFolder, vimtype, True)
    for c in container.view:
        if c.name == name:
            return c
    return None


def get_customer_attribute_key(vc_connection, attr_name):
    fields = vc_connection.content.customFieldsManager.field
    for field in fields:
        if field.name == attr_name:
            return field.key


def get_customer_attribute_value(vc_connection, vm_name, attr_name):
    attr_key = get_customer_attribute_key(vc_connection, attr_name)
    content = vc_connection.RetrieveContent()
    vm = get_obj(content, [vim.VirtualMachine], vm_name)
    if vm.customValue:
        for customer in vm.customValue:
            if customer.key == attr_key:
                return customer.value
    return None


def add_customer_attribute(vc_connection, vm_name, attr_name, attr_value):
    content = vc_connection.RetrieveContent()
    vm = get_obj(content, [vim.VirtualMachine], vm_name)
    attr_key = get_customer_attribute_key(vc_connection, attr_name)
    if not attr_key:
        attr_key = content.customFieldsManager.AddFieldDefinition(name=attr_name, moType=vim.VirtualMachine)
        attr_key = attr_key.key
    vc_connection.content.customFieldsManager.SetField(entity=vm, key=attr_key, value=attr_value)


def clone_vm_template(vc_connection, src_name, new_name, folder_name, datacenter_name, cluster_name, datastore_name, create_template=False):

    content = vc_connection.RetrieveContent()
    src_object = get_obj(content, [vim.VirtualMachine], src_name)
    datacenter = get_obj(content, [vim.Datacenter], datacenter_name)
    cluster = get_obj(content, [vim.ClusterComputeResource], cluster_name)
    datastore = get_obj(content, [vim.Datastore], datastore_name)
    vm_folder = get_obj(content, [vim.Folder], folder_name)

    if not vm_folder:
        vm_folder = datacenter.vmFolder

    # Relocation spec
    relospec = vim.vm.RelocateSpec()
    relospec.datastore = datastore
    relospec.pool = cluster.resourcePool

    # Clone spec
    clonespec = vim.vm.CloneSpec()
    clonespec.location = relospec
    clonespec.template = create_template

    print("Cloning %s to %s ......" % (src_name, new_name))
    task = src_object.Clone(folder=vm_folder, name=new_name, spec=clonespec)
    time.sleep(5)
    while task.info.state == vim.TaskInfo.State.running:
        time.sleep(2)

    if task.info.state != vim.TaskInfo.State.success:
        raise Exception('Clone task failed')
    print("Cloning is successful.")

    new_uuid = str(uuid.uuid4())
    print('Add UUID %s to the new object.' % new_uuid)
    add_customer_attribute(vc_connection, new_name, 'UUID', new_uuid)

    parent_uuid = get_customer_attribute_value(vc_connection, src_name, 'UUID')
    print('Get UUID of source object: %s' % parent_uuid)
    if not parent_uuid:
        parent_uuid = 'None'
    print('Add PARENT_UUID %s on new object %s' % (parent_uuid, new_name))
    add_customer_attribute(vc_connection, new_name, 'PARENT_UUID', parent_uuid)


def collect_vm_info(vc_connection):
    vms_info = {}
    uuid_key = get_customer_attribute_key(vc_connection, 'UUID')
    parent_uuid_key = get_customer_attribute_key(vc_connection, 'PARENT_UUID')

    content = vc_connection.RetrieveContent()
    vms = content.viewManager.CreateContainerView(content.rootFolder, [vim.VirtualMachine], True).view
    for vm in vms:
        vm_uuid, parent_uuid = None, None
        for customer in vm.customValue:
            if customer.key == uuid_key:
                vm_uuid = customer.value
            if customer.key == parent_uuid_key:
                parent_uuid = customer.value
        if vm_uuid:
            vms_info[vm_uuid] = {
                'name': vm.name,
                'parent_uuid': parent_uuid,
                'type': 'T' if vm.config.template else 'V'
            }
    return vms_info


def create_node(vm_uuid, vms_info):
    current = vms_info[vm_uuid]
    if 'node' not in current:
        node_name = ' ' + vms_info[vm_uuid]['name'] + ':%s' % vm_uuid.split('-')[0] \
            + '(%s) ' % vms_info[vm_uuid]['type']
        if current['parent_uuid'] is None:
            current['node'] = Node(node_name)
        else:
            parent_uuid = current['parent_uuid']
            parent = vms_info[parent_uuid]
            if 'node' not in parent:
                create_node(parent_uuid, vms_info)
            current['node'] = Node(node_name, parent['node'])


if __name__ == '__main__':
    parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('-c', '--clone', required=False, action='store_true', help='Clone VM or template')
    parser.add_argument('-t', '--tree', required=False, action='store_true', help='Display inheritance tree')

    parser.add_argument('--src_name', required=False, type=str, help='Specify the name of the source VM or template')
    parser.add_argument('--new_name', required=False, type=str, help='Specify th name of the target VM or template')
    parser.add_argument('--folder', required=False, type=str, help='Specify th name of the folder to put new VM or template')
    parser.add_argument('--datacenter', required=False, type=str, help='Specify th name of the datacenter to put new VM or template')
    parser.add_argument('--cluster', required=False, type=str, help='Specify th name of the cluster to put new VM or template')
    parser.add_argument('--datastore', required=False, type=str, help='Specify th name of datastore to put new VM or template')
    parser.add_argument('--create_template', required=False, choices=['True', 'False'], type=str, help='Specify if the target is template or not')

    args = parser.parse_args()

    vc_connection = get_smart_connetion(
                    os.environ.get('vcenter_host'),
                    os.environ.get('vcenter_user'),
                    os.environ.get('vcenter_password')
    )

    if args.clone:
        clone_vm_template(
            vc_connection=vc_connection,
            src_name=args.src_name,
            new_name=args.new_name,
            folder_name=args.folder,
            datacenter_name=args.datacenter,
            cluster_name=args.cluster,
            datastore_name=args.datastore,
            create_template=bool(args.create_template)
        )
    elif args.tree:

        vms_info = collect_vm_info(vc_connection)
        root_objects = []
        for vm_uuid in vms_info:
            if vms_info[vm_uuid]['parent_uuid'] is None:
                root_objects.append(vm_uuid)
            create_node(vm_uuid, vms_info)
        for object_uuid in root_objects:
            print_tree(vms_info[object_uuid]['node'], horizontal=True)

2. Use IBM Cloud for VMware Solutions REST API to get the credential of your vCenter Server instances

Get the IBM Cloud for VMware Solutions REST API, which requires an IBM Cloud IAM access token in the request header for authentication. The IAM token can be generated from your IBM Cloud API key.

In IBM Cloud for VMware Solutions, vCenter Server instances are identified by the instance ID. The following API is used to list all vCenter Server instances you have and show the basic information of each instance which includes instance ID:

GET /v1/vcenters

By giving the instance UUID, the following API returns the detailed information of a vCenter Server instance, which includes the vCenter credentials:

GET /v1/vcenters/{instance_id}

A helper is implemented in the Python module ic4v_helper.py to use the IBM Cloud API key to return the list of vCenter server instances with instance ID, name and status, and use the instance ID to retrieve the vCenter host and credential information of the given vCenter Server instance:

# file name: ic4v_helper.py

import os
import requests
import argparse
from prettytable import PrettyTable


def authenticate():
    headers = {'Accept': 'application/json'}
    params = {
        'grant_type': 'urn:ibm:params:oauth:grant-type:apikey',
        'apikey': os.environ.get('api_key')
    }
    auth_url = 'https://iam.cloud.ibm.com/identity/token'
    resp = requests.post(auth_url, data=params, headers=headers)
    if resp.status_code != 200:
        raise Exception('Failed to generate IAM token')

    json = resp.json()
    headers['Authorization'] = 'Bearer %s' % json['access_token']
    headers['X-Auth-Refresh-Token'] = json['refresh_token']
    return headers


def list_vcenters():
    ic4v_uri = 'https://api.vmware-solutions.cloud.ibm.com/v1/vcenters'
    resp = requests.get(ic4v_uri, headers=authenticate())
    if resp.status_code != 200:
        raise Exception('Failed to generate IAM token')
    return resp.json()


def get_vcenter_credential(instance_uuid):
    ic4v_uri = 'https://api.vmware-solutions.cloud.ibm.com/v1/vcenters/%s' % instance_uuid
    resp = requests.get(ic4v_uri, headers=authenticate())
    if resp.status_code != 200:
        raise Exception('Failed to generate IAM token')
    vcenter = resp.json()
    endpoints = vcenter.get('endpoints', [])
    for endpoint in endpoints:
        if endpoint['type'] == 'vCenter/PSC':
            for c in endpoint['credentials']:
                if c['type'] == 'ADMIN':
                    return endpoint['ip_address'], c['user'], c['password']


if __name__ == '__main__':
    parser = argparse.ArgumentParser(formatter_class=argparse.RawDescriptionHelpFormatter)
    parser.add_argument('-l', '--list', required=False, action='store_true', help='List all vCenter instances')
    parser.add_argument('-cr', '--credential', required=False, type=str, help='Get the vCenter host and credentials')

    args = parser.parse_args()
    if args.list:
        vcenters = list_vcenters()
        column_name, column_uuid, column_status = [], [], []
        for vcenter in vcenters:
            column_name.append(vcenter['name'])
            column_uuid.append(vcenter['id'])
            column_status.append(vcenter['status'])
        instances = PrettyTable()
        instances.add_column('Name', column_name)
        instances.add_column('Instance ID', column_uuid)
        instances.add_column('Status', column_status)
        instances.header = True
        print(instances)
    elif args.credential:
        host, user, password = get_vcenter_credential(args.credential)
        print('vcenter_host="%s"' % host)
        print('vcenter_user="%s"' % user)
        print('vcenter_password="%s"' % password)

3. Use the tool on your vCenter Server instances

This step shows you how to use the tool module vm_relation_tool.py and the helper module ic4v_helper.py to clone virtual machines and templates, and create the inheritance trees in a vCenter Server instance of IBM Cloud for VMware Solutions.

• Get the vCenter credential of your vCenter server instance. The ic4v_helper.py reads your API key from environment variable, set the environment variable before running it:

export api_key=<your IBM Cloud API key>

Use the command to get the vCenter Server instances:

python ic4v_helper.py -l

Following is the example of the result.

Copy the instance ID of one instance at the status ReadyToUse, and run the command to get the vCenter host and credential information:

python ic4v_helper.py -cr <instance id>

• Clone new virtual machines or templates. The vm_relation_tool.py tool must run on the machine that can connect to vCenter on port 443, and Python3, pyVmomi, and pptree must be installed on the machine. Before running vm_relation_tool.py, set the environment variables with the vCenter host and credential information you obtained from ic4v_helper.py to specify the vCenter you want to use:

export vcenter_host={vCenter host name}
export vcenter_user={vCenter HTTPS user name}
export vcenter_password={vCenter HTTPS password}

Use the command to clone a virtual machine or template:

python vm_relation_tool.py -c --src_name <source VM/template name> --new_name <target VM/template name> --folder <foler for target VM/template> --datacenter <datacenter name> --cluster <cluster name> --datastore <datastore name> --create_template <True if target is template>

Here is the example of running the command to clone a newa-rhel75 virtual machine from template rhel75:

python vm_relation_tool.py -c --src_name rhel75 --new_name a-rhel75 --folder test-template --cluster cluster1--datastore datastore1 --create_template False

The new virtual machine showed up on vCenter with the UUID and PARENT_UUID attached after the command succeeded.

• Generate and display inheritance trees. Use the command to generate and display the inheritance trees:

 python vm_relation_tool.py --tree

Here is the example of the command result. It generated the inheritance trees of the virtual machines and templates appearing in the vCenter view at the beginning of this tutorial.

Extension considerations

• The Python code in the tool vm_relation_tool.py only uses the core feature of pyVmomi cloning method to demonstrate the basic scenario of cloning. However, this method supports a variety of customizations on the cloned virtual machine or template, from CPU, memory, disk to network adaptor and CD/DVD drive. The pyVmomi community includes more resources, which can be used for reference to extend above code to support more customized scenarios of cloning virtual machine and template.
• It is also encouraged to integrate the idea and code in this tutorial into the standard image automation pipelines or tools, like HashiCorp Packer, which provides a plugins mechanism to extend its core functions.

Summary

After completing this tutorial, you should be able to write a tool for cloning virtual machines or templates by adding Custom Attributes to track inheritance relationships, and create inheritance trees from tracked virtual machines and templates. You should also know the APIs of getting your vCenter Server instances in IBM Cloud for VMWare solutions platform, and use the tool on your vCenter Server instances. This tool will be helpful if you are a virtual machine templates developer and will help you sort out the changes made in a large number of templates.