Deployment steps: IBM VM Recovery Manager HADR cluster

Overview

The IBM VM Recovery Manager family provides high availability (HA) and disaster recovery (DR) solutions for virtual machines (VMs) within a site and across sites using a single orchestrator. In case of down time or planned outage, user can migrate a VM from one host (server) to another host within a home site (for partial failure) or backup site (for complete failure). This article explains the deployment of a VM Recovery Manager HADR cluster using a single KSYS node.

VM Recovery Manager HADR for Power Systems support different types of failures within a site and across sites. Within a site, failures can occur at application level, VM level, or host level. In case of failures within a site, the VM migrates automatically. But if the failure is across sites, you must initiate a move operation. Across site includes planned and unplanned move at site and host group level, here user will trigger move in case of any failure. VM Recovery Manager HADR uses shared storage pool (SSP) cluster formed within source site Virtual I/O Server (VIOS) instances for VM and host monitoring to provide high availability solution.

Figure 1. HADR configuration handled by a single KSYS node

Configuration prerequisite

The IBM VM Recovery Manager HADR cluster requires the following prerequisites with respect to hardware configuration:

• For a HADR cluster, you should have at least two hosts on the home site and a minimum of one host on the backup site.
• Home site hosts should have a minimum of two VIOS instances.
• Home site VIOS instances should have at least two shared disks between them for an SSP cluster and Cluster Aware AIX (CAA) volume group for repository. The size of the shared disks must be greater than or equal to 5 GB.
• All managed VMs should have disks on the source and target storages and replication should exist between the disks.

• Include the /opt/IBM/ksys path to the root directory so that you need not specify this path every time you run the ksysmgr command. To include this path, run the following command in the KSYS node:

  export PATH=$PATH:/opt/IBM/ksys

Important file sets to be installed on VM and VIOS

A VM agent and host monitor file sets are necessary for VM Recovery Manager HADR cluster. The VM agent file sets are included with KSYS and they are installed on virtual machines. Host monitor file sets are installed on VIOS.

File sets and commands for VM

To monitor the VMs, install the VM agent on the VMs and ensure that the VM agent is running. Run the following commands to start, stop, and query the VM agent.

• Start a VM agent: ksysvmmgr start
• Stop a VM agent: ksysvmmgr stop
• Query a VM agent: ksysvmmgr status

File sets and commands for VIOS

Before running discovery, install the host monitor file sets on all home site VIOS instances. A host monitor daemon (hm_daemon) is a specific daemon designed for VM Recovery Manager. At the start of a cluster creation, hm_daemon should be in the inoperative state

To check hm_daemon, log in to the VIOS instances as a root user and run the following command:
lssrc -s ksys_hsmon

Configuration steps

Perform the following steps to configure a VM Recovery Manager HADR cluster:

1. Add a KSYS cluster.
2. Add the home and backup sites.
3. Add the Hardware Management Console (HMC) on the home and backup sites.
4. Add the hosts.
5. Pair the hosts if the host_group type is symmetric.
6. Add a storage agent for both the sites.
7. Add a host group.
8. Add ha_disk and repo_disk of VIOS instances, if VIOS does not have an SSP cluster.
9. Run discovery.
10. Run verify.

Command to create, verify, sync, and query a KSYS cluster

This section provides information about how to create, verify, sync and query a KSYS cluster.

How to create a KSYS cluster

This command creates a cluster based on the type specified. VM Recovery Manager supports three types of clusters: HA, DR, and HADR. Refer to the following basic format for using the cluster creation command:

ksysmgr add ksyscluster <cluster_name> ksysnodes=<ksys node hostname> type=HADR

Example: ksysmgr add ksyscluster HADR_CLUSTER ksysnodes=hostname type=HADR

How to verify a KSYS cluster

This command verifies whether the cluster is created or not.

ksysmgr verify ksyscluster <cluster_name>

Example:ksysmgr verify ksyscluster HADR_CLUSTER

How to sync a KSYS cluster

This sync command starts the KSYS daemon on KSYS node.

ksysmgr sync ksyscluster <cluster_name>

Example: ksysmgr sync ksyscluster HADR_CLUSTER

How to create a KSYS cluster with the sync option

This single command creates the KSYS cluster, verifies it, and starts the daemon on KSYS node.

ksysmgr add ksyscluster <name> ksysnodes=<ksys node name> sync=yes type=HADR

Example: ksysmgr add ksyscluster HADR_CLUSTER type=HADR ksysnodes=hostname sync=yes

How to query a KSYS cluster

This command queries for KSYS cluster information and displays details such as cluster name, cluster type, and state of the cluster. Here clustername is optional.

ksysmgr query ksyscluster [clustername]

Commands to add, modify, and delete a site

This section provides information on how to add, query, modify, and delete a site from a KSYS cluster.

How to add a site to a KSYS cluster

There are two type of sites: Home site and backup site. The following command adds a site by providing the site type to a KSYS cluster.

ksysmgr add site <sitename> sitetype=<home|backup>

Example: ksysmgr add site HomeSite sitetype=home

Note: Users should first add the home site.

How to query for site details from a KSYS cluster

This command queries and provides output for the site that is configured to the KSYS cluster. Here sitename is optional.

ksysmgr query site [sitename]

How to modify site details in a KSYS cluster

This command allows you to modify a site’s attributes (such as memory_capacity, cpu_capacity, skip_resource_check, skip_power_on, and proactiveha) and its network settings.

ksysmgr modify site <sitename> [skip_resource_check=<yes|no>]

Example:ksysmgr modify site HomeSite skip_resource_check=yes

How to delete a site from a KSYS cluster

This command deletes a site from the KSYS cluster.

ksysmgr delete site <sitename>

Example: ksysmgr delete site HomeSite

Commands to add, query, modify, and delete HMC

This section provides information on how to add, modify, and delete HMC from a KSYS cluster.

How to add HMC to a KSYS cluster

This command adds a HMC by using a username, password, HMC host name or IP, and sitename.

ksysmgr add hmc <hmcname> login=<username> [password=<password>] hostname|ip=<hostname|ip> site=<sitename>

Example: ksysmgr add hmc hmc1 login=****** password=****** hostname=XX.XX.XX.XX site=HomeSite

How to query for HMC details from a KSYS cluster

This command queries and gives output for HMC which is configured to the KSYS cluster. Here hmcname is optional.

ksysmgr query hmc [hmcname]

How to modify HMC details in a KSYS cluster

This command allows to modify HMC hostname, IP, login, and password.

ksysmgr modify hmc <hmcname> [login|password|hostname|ip=username|pwd|hostname|ip]

Example: ksysmgr modify hmc hmc1 hostname=hmc.aus.stglabs.ibm.com

How to delete HMC from a KSYS cluster

This command deletes the HMC from the KSYS cluster.

ksysmgr delete hmc <hmcname>

Example: ksysmgr delete hmc hmc1

Commands to add a host to a KSYS cluster

Use the following information to find the syntax for adding a host to a KSYS cluster.

How to add a host to a KSYS cluster

This command adds a host to the KSYS cluster. If we have more than one host with the same name, use the Universally Unique Identifier (UUID), host name, or the IP of a host.

Ksysmgr add host <hostname> site=<sitename>

Example: ksysmgr add host host11 site=HomeSite

How to query a host in a KSYS cluster

This command queries and provides the details for a host Here hostname is optional. If the host name is not provided, the command queries all the hosts of the KSYS cluster.

ksysmgr query host <hostname>

How to modify the host attributes in a KSYS cluster

This command modifies the host attributes to the respective values. It allows to modify attributes such as skip_power_on, ha_monitor, proactiveha, vm_failure_detection_speed and network settings at the host level.

ksysmgr modify host <hostname> [ha_monitor=<enable | disable>]

Example: ksysmgr modify host host11 ha_monitor=enable

How to delete a host from a KSYS cluster

This command deletes the host from the KSYS cluster.

ksysmgr delete host <hostname>

Example: ksysmgr delete host host11

Command to pair or unpair hosts

Use the following information to find the syntax for pairing or unpairing hosts. It allows only 1:1 pairing. If users want to create a symmetric host group then hosts need to be paired, for an asymmetric host group, a host pair is not needed.

How to pair host

This command will create a pairing between home site host and backup site host

ksysmgr pair host <hostname> pair=<name>

Example: ksysmgr pair host host11 pair=host21

How to unpair hosts

This command removes pairing between the home site host and the backup site host

ksysmgr pair host <hostname> pair=none

Example: ksysmgr pair host host11 pair=none

Commands to query for VIOS and VM details from a KSYS cluster

Use the following information to find the syntax for querying VIOS and VM from a KSYS cluster.

How to query for VIOS details from a KSYS cluster

This command queries and provides the details of a VIOS instance. Here, viosname is optional.

Note: A cluster can have multiple VIOS instances. If you run a command with viosname,will you can find details of that specific VIOS instance.

ksysmgr query vios [viosname]

How to query for VM details from a KSYS cluster

This command queries and provides the details of a VM. Here vmname is optional.

Note: A cluster can have multiple VMs. If you run a command with vmname, you can find details specific to that VM.

ksysmgr query vm [vmname]

Commands to manage and unmanage VIOS and VM

Use the following information to find the syntax for managing and unmanaging a VIOS instance or a VM.

How to unmanage VIOS instances using ksysmgr

This command unmanages VIOS instances. The following example unmanages the vios_name1 and vios_name2 instances.

ksysmgr unmanage vios <vios_name1,vios_name2..n> Example: ksysmgr unmanage vios vios_name1,vios_name2

How to manage VIOS using ksysmgr

This command manages VIOS instances. The following example manages the vios_name1 and vios_name2 instances.

ksysmgr manage vios <vios_name1,vios_name2..n>

Example: ksysmgr manage vios vios_name1,vios_name2

How to unmanage a VM using ksysmgr

This command unmanages the VMs. The following example unmanages the VMname1 and VMname2 VMs.

ksysmgr unmanage vm <VMname1,VMname2..n>

Example: ksysmgr unmanage vm VMname1,VMname2

How to manage a VM using ksysmgr

This command manages the VMs. The following example manages the VMname1 and VMname2 VMs.

ksysmgr manage vm <VMname1,VMname2..n >

Example: ksysmgr manage vm VMname1,VMname2

Command to modify VM values

Use the following information to find the syntax for modifying VM values. This command allows you to modify the attributes at the VM level that includes homehost, DrTargetHost, priority, skip_power_on, ha_monitor, and vm_failure_detection_speed.

ksysmgr modify vm <VMname1,VMname2..n > ha_monitor=enable

Example: ksysmgr modify vm sock003 ha_monitor=enable

Commands to add, query, modify, and delete a storage agent in a KSYS cluster

Use the following information to find the syntax for adding, querying, modifying, and deleting a storage agent to a KSYS cluster.

How to add a storage agent to a KSYS cluster

This command adds a storage agent to the KSYS cluster. Addition of a storage agent varies based on the type of storage.

ksysmgr add storage_agent <storage_agent_name> hostname|ip =<hostname[,hostname2,hostname3]|ip[,ip2,ip3]> site=<sitename> storagetype=<type> serialnumber=<number> | clusterid=<number> | instance=<instance_number> [login=<username>] [password=<password>]

Example: ksysmgr add storage_agent STA1 login=****** password=****** site=HomeSite storagetype=ds8k hostname=XX.XX.XX.XX

How to query a storage agent in a KSYS cluster

This command queries and provides the details for a storage agent. Here, storage_agent_name is optional. If no name is mentioned in the command, ksysmgr queries all the storage agent details.

ksysmgr query storage_agent [<storage_agent_name>]

How to modify the storage_agent attributes in a KSYS cluster

This command modifies the storage_agent attribute values. It allows you to modify storage_agent_name, hostname, ip, login, password, and drinstance number.

ksysmgr modify storage_agent <storage_agent_name> [hostname=<hostname[,hostname2,hostname3]>]

Example: ksysmgr modify storage_agent STA1 hostname=STA1_New

How to delete a storage agent from a KSYS cluster

This command deletes the storage agent from the KSYS cluster. Before deleting a storage agent, make sure that disk_pair and disk_group is empty.

ksysmgr delete storage_agent <storage_agent_name>

Example: ksysmgr delete storage_agent STA1

Commands to query an SSP disk from a KSYS cluster

Use the following information to find the syntax for querying SSP disks from the VIOS instances. Two disks are needed, one for ha_disk and one for repo_disk. If you have user-defined SSP cluster in a setup, then this step is not required.

How to query an SSP disk from a managed VIOS

This command queries for free shared disks at VIOS, host, and host group level.

ksysmgr query viodisk <vios=<viosname[,viosname2,...] || hosts=<hostname1[,hostname2,...] || host_group=<host_group_name> >

Example: ksysmgr q viodisk vios=vios1,vios2

Commands to create, query, modify, and delete host groups from a KSYS cluster

Use the following information to find the syntax for adding, querying, modifying, and deleting a host group from a KSYS cluster.

How to add a host group to a KSYS cluster

This command adds a host group with host11 and host12 as home site hosts, pair of host11 and host12 as target site hosts, Rdisk as the repo_disk, and Hdisk as the ha_disk to the KSYS cluster. If mxn_pairing=yes KSYS will create a asymmetric type of HG, if mxn_pairing=no or mxn_pairing attribute is missing then KSYS will create a symmetric type of the host group. In case of an asymmetric host group, users must provide the target site host in the hosts list.

ksysmgr add hg <hg_name> hosts=<host1,host2,..> repo_disk=<ID> ha_disk=<ID> mxn_pairing=<yes>

Example: ksysmgr add hg HG1 hosts=host11,host12 repo_disk=Rdisk ha_disk=Hdisk

Note: repo_disk, ha_disk and mxn_pairing is optional while adding a host group.

How to query a host group using ksysmgr

This command queries and provides the output for HG1.

ksysmgr query host_group [host_group]

How to modify host group attributes using ksysmgr

The host_group attributes can be modified using flags add, remove and options.

ksysmgr modify host_group <name> add hosts=<host13,host14>
ksysmgr modify host_group <name> remove hosts=<host13,host14>
ksysmgr modify host_group <name> options ha_monitor=<enable>

Example: ksysmgr modify hg HG1 add hosts=host3,host4

With the help of attribute restart_policy at the host group level, users can change how VM migrates during failure within a site. If restart_policy=auto VM migrates automatically but if restart_policy=advisory_mode then users will get the notification and event in events.log and users have to trigger the move operation.

Commands to add, query, modify, and delete remote notify

Use the following information to find the syntax to add, query, modify, and delete remote notify.

ksysmgr add notify user= <username> contact=<contact>
ksysmgr query notify [contact]
ksysmgr modify notify oldcontact= <user_name|user_email> newcontact=<user_name|user_email>
ksysmgr delete notify user=<user_name>

Commands to perform KSYS discovery and Verify

Use the following information to find the syntax for performing discovery and verification.

How to perform discovery

These commands perform discovery at the host group and site levels. During discovery, KSYS creates an SSP cluster in the VIOS instances, disk_pair and disk_group at the KSYS level and consistency group at the storage level. If user want to monitor VMs, VIOS instances, and hosts then ha_monitor should be enabled at the system, host group, host, and VM levels. By default, ha_monitor is enabled at the system, host group, and host levels but at the VM level it will be disabled.

ksysmgr discover hg <host_group_name> ksysmgr discover site <sitename>

How to perform verify

These commands perform verification at host group and site level.

ksysmgr verify hg <host_group_name> ksysmgr verify site <sitename>

How to perform discovery and verification in a single command

To perform discovery and verification in a single command at host group level and site level, run:

ksysmgr discover hg <host_group_name> verify=yes
ksysmgr discover site <sitename> verify=yes

Example: Output of discovery and verification using a single command

(0) root @ ksys204: /
# ksysmgr discover hg HGTest verify=true
Running discovery on Host_group HGTest, this may take few minutes...
        Storage state synchronization has started for Host_group HGTest
        Storage state synchronization has completed for Host_group HGTest
        Existing HA trunk adapter found for VIOS raguv1
        Existing HA trunk adapter found for VIOS raguv2
        Existing HA trunk adapter found for VIOS sockv1
        Existing HA trunk adapter found for VIOS sockv2
        Preparing VIOS in ragu-9119-MME-SN106CDC7 for HA management
        VIOS in ragu-9119-MME-SN106CDC7 prepared for HA management
        Preparing VIOS in sock_8286-42A-2182C5V for HA management
        VIOS in sock_8286-42A-2182C5V prepared for HA management
        Preparing VM sock003 in Host ragu-9119-MME-SN106CDC7 for HA management
        VM sock003 in Host ragu-9119-MME-SN106CDC7 Prepared for HA management
        Existing first HA client adapter found for VM sock003
        Existing second HA client adapter found for VM sock003
        Discovery has started for VM sock003
        Configuration information retrieval started for VM sock003
        HA monitoring for VM sock003 started successfully
        Configuration information retrieval completed for VM sock003
        Storage information retrieval from VIOS started for VM sock003
        Storage information retrieval from VIOS completed for VM sock003
        Discovery for VM sock003 is complete
        VM monitor state has moved to 'STARTED' for VM sock003
        Disk Group creation on storage subsystem started for Host_group HGTest
        Disk Group creation on storage subsystem completed for Host_group HGTest
Discovery has finished for HGTest
1 out of 1 managed VMs have been successfully discovered

Host_group verification started for HGTest
        sock003 HA verification has started
        sock003 HA verification has completed
        shoe_8286-42A-2100AAW verification has started
        shoe_8286-42A-2100AAW verification has completed
        ziti_9119-MME-106CD27 verification has started
        ziti_9119-MME-106CD27 verification has completed
        sock003 DR verification has started
        sock003 DR verification has completed
        Disk Group verification on storage subsystem started for Host_group HGTest
        Disk Group verification on storage subsystem completed for Host_group HGTest
Verification has finished for HGTest
1 out of 1 VMs have been successfully verified

Commands to query disk_pair, disk_group, and an SSP cluster

Use the following information to find the syntax for querying disk_pair, disk_group, and an SSP cluster.

How to query a disk_pair

This command queries pairing of all the disks. By default, disk_pair will be empty before first discovery is run.

ksysmgr q disk_pair

How to query a disk_group

This command queries disk groups created using disk pairs. By default disk_group will be empty before first discovery is run. Here disk_group_name is optional.

Note: Passing disk_group_name in the command shows the details of a specific disk group only.

ksysmgr q disk_group <disk_group_name>

How to query an SSP cluster in VIOS

If a user-defined cluster does not exist in the VIOS instances, then KSYS creates a KSYS-defined SSP cluster. To query an SSP cluster, log in as padmin and run: cluster -status

Commands to perform dr_test discovery and verification

Use the following information for performing dr_test discovery.

How to perform dr_test discovery

These commands perform dr_test discovery at the host group and site levels. For dr_test discovery, user should have a cloned disk for all managed VMs on the target storage.

ksysmgr discover hg <host_group_name> dr_test=yes ksysmgr discover site <sitename> dr_test=yes

How to perform dr_test verification

You can perform dr_test verification at the host group and site levels.

ksysmgr verify hg <host_group_name>
ksysmgr verify site <sitename> dr_test=yes

How to perform discovery and verify in single command

You can perform dr_test discovery and verification in single command at host group level and site level.

ksysmgr discover hg <host_group_name> verify=yes dr_test=yes
ksysmgr discover site <sitename> verify=yes dr_test=yes

Commands to monitor detailed discovery and quick discovery

Use the following information to monitor quick discovery and detailed discovery.

How to monitor a detailed and quick discovery run by KSYS in the background

This command monitors the status of detailed and quick discovery run by the scheduler. Detailed discovery is triggered automatically once in 24 hours. Quick discovery is triggered every hour. You can modify the values of detailed and quick discovery using:

ksysmgr modify system auto_discovery_time|quick_discovery_interval|quick_discovery

How to see KSYS operation status, if KSYS node console is not open

The following command shows the status of the current system. If user triggers a KSYS operation such as discovery, verification, move, and so on, the console gets terminated. Users can open a new console and see the current status of the operation using this command.

ksysmgr q system status monitor=yes

How to check for KSYS errors if discovery or verification fails

The following command displays an error message on the console stating why discovery or verification has failed.

ksysmgr q system status

Commands to collect, query, and restore snapshot

KSYS supports two types of snapshots: CLUSTER and DETAILED snapshot. If type is not mentioned, then the default is DETAILED snapshot. Default path for collecting snapshot is /var/ksys/snapshots/.

How to collect a CLUSTER snapshot using ksysmgr

This command captures a snapshot of type CLUSTER and stores it at the given file path.

ksysmgr add snapshot filepath=<path> type=CLUSTER

Example: ksysmgr add snapshot filepath=/var/ksys/snapshots/snap type=CLUSTER

How to collect a DETAILED snapshot using ksysmgr

This command captures a snapshot of type DETAILED and stores it at the given file path.

ksysmgr add snapshot filepath=<path> type=DETAILED

Example: ksysmgr add snapshot filepath=/var/ksys/snapshots/snap type=DETAILED

How to query a snapshot

Use the following information to find the syntax for querying snapshots. Here file path is optional. If file path is not given, then KSYS queries the snaps collected in the default path.

ksysmgr query snapshot filepath=<filepath>

How to restore snapshot for CLUSTER and DETAILED snapshots

Use the following command to restore a snapshot.

ksysmgr restore snapshot filepath=<filepath>

Example: ksysmgr restore snapshot filepath=/var/ksys/snapshots/snap

Commands to delete a KSYS cluster

Use the following information to delete a KSYS cluster.

How to delete a KSYS cluster

This command deletes the KSYS cluster from the KSYS node. If the operation fails, an error message is displayed.

Ksysmgr delete ksyscluster <cluster_name>

Example: ksysmgr delete ksyscluster HADR_CLUSTER

How to delete a KSYS cluster with the force option

The -f option in this command deletes the KSYS cluster from the KSYS node forcefully.

ksysmgr -f delete ksyscluster <cluster_name>

KSYS logs path or location for troubleshooting

This section provides information about the path of the KSYS logs for troubleshooting.

Where can you find ksysmgr logs

ksysmgr logs are available at the following path:

/var/ksys/log/ksysmgr.log

Where ksysmgr execution commands are logged

ksysmgr execution commands are logged at the following path:

/var/ksys/log/ksysmgr.oplog

Where KSYS traces will be logged

KSYS traces will be logged at the following path:

/var/ct/<cluster_name>/log/mc/IBM.VMR

Which tool is used to extract KSYS traces

The rpttr tool is used to extract the traces.

How to view live traces and extract KSYS log using rpttr

KSYS cluster complete traces and KSYS logs can be extracted in a user-defined file using rpttr .

rpttr -fodict trace.ksys.* > ksys.txt

Using the -f option with rpttr shows live traces in run time.

rpttr -fodict trace.ksys.*

Where are the storage traces logged

The storage traces are logged at the following path:

/var/ksys/log/ksys_<storagetype>.log

Example: /var/ksys/log/ksys_ds8k.log

Where are the events logged

In a KSYS cluster there two type of events, namely critical events and operational events. Critical events show all the failures, errors in a cluster resource such as HMC, host, storage agent, VM, and so on. Operational events show events related to success and failure of operations such as discovery, verification, move, clean up, restart, recovery, and so on.

/var/ksys/events.log /var/ksys/opevents.log

How to collect snaps of VM, KSYS, host monitor, and VIOS

VM: ksysvmmgr snap
Host monitor/VIOS: snap -a
KSYS: snap vmsnap

How to collect HMC logs

You can collect HMC logs using the following command:

pedbg -c -q 4

Note: To collect HMC logs, log in to HMC using hscpe user credentials.

Conclusion

This article can help users to create and configure a HADR cluster and provide a suitable environment with HA and DR capability using a single KSYS node in the ksysmgr CLI.